AI Agents Now Build Reliable Web Scrapers Without Human Intervention
In brief
- AI agents are now capable of building reliable web scrapers through a new framework that avoids common pitfalls like broken selectors and schema mismatches.
- This breakthrough comes from experiments on 138 tasks, where the system consistently produced accurate results by using a structured approach involving JSON configurations and six-type collector taxonomy.
- The key innovation lies in shifting LLM output from free-form code to typed JSON structures, which ensures deterministic and verifiable execution paths.
- On 80 independently verified tasks, this method achieved zero execution-stage errors and the lowest average wall-clock time, prioritizing reusable and reliable data collection over initial quality.
- This advancement marks a significant step toward more dependable AI-driven web scraping tools, enabling developers to streamline repetitive data extraction processes with fewer manual interventions.
- The framework's ability to handle repeated tasks efficiently could pave the way for broader adoption in industries reliant on real-time data collection.
Terms in this brief
- JSON configurations
- A structured format for storing data in key-value pairs, used here to define web scraping tasks clearly and consistently.
- Collector taxonomy
- A classification system for different types of data collection methods, helping organize and standardize how web scrapers operate.
Read full story at arXiv CS.AI →
More briefs
New Gemini 3.7 Flash Model Released
Google introduced Gemini 3.7 Flash, a more intelligent model for coding and agents. This new model comes just three weeks after the previous version. Gemini 3.7 Flash has many improvements, including better performance in coding tasks and web development. It can generate more functional layouts and feature-complete apps in fewer prompts. The model also shows high design adherence and parity based on a reference input. Gemini 3.7 Flash will be available at half the original price of the previous model. It delivers substantial improvements across software engineering and knowledge work. The future of coding and agents will likely be shaped by this new model.
Drones Compete in DARPA Lift Challenge
Drones recently competed in the DARPA Lift Challenge. The challenge tested the drones' ability to lift heavy objects. The challenge included many unique drones. Some drones were able to lift objects that weighed over 50 pounds. This matters because it could help drones be used in search and rescue missions. New drone technologies are being developed every day and will likely be used in many industries soon.
Anthropic Explains Chatbot Watermarking
Anthropic published a blog post to answer questions about watermarking its chatbot Claude. The company will add a hidden pattern to the text generated by Claude. The watermarking is required by the EU AI Act’s Transparency Code. This code makes AI companies identify AI-generated content. Anthropic said the watermark will not change the quality of Claude’s output. The company plans to release a watermark detection API. Someone can hide the watermark by completely rewriting the text. The future of AI-generated content will be closely watched as more companies comply with the new regulations.
Google Allows Users to Remove Visible Watermark from AI Generations
Google will now allow users to remove a visible watermark from its AI generations, including images, videos, and songs. This change will affect models like Nano Banana, Omni, and Lyria. The visible watermark was making content less useful for professional and creative work. But Google still wants to identify AI-generated content. The company will keep using invisible SynthID watermarks and metadata for transparency. Over 100 million users will have access to this new feature. Users will be able to turn the visible watermark on or off in the coming days. They can go to Settings to make this change. Google will continue to update its AI features.
AI Satellites to Track Wildfires
Researchers are testing a system that uses artificial intelligence to detect wildfires and reposition satellites for more frequent data. This system could help firefighters detect and fight wildfires sooner. The system could capture more information about wildfires, especially in remote areas. Satellites can detect fires and move to get better views. Current systems use fixed orbits, but this new system allows satellites to reposition themselves. Firefighters will get critical information faster.