AI Agents Now Build Reliable Web Scrapers Without Human Intervention
In brief
- AI agents are now capable of building reliable web scrapers through a new framework that avoids common pitfalls like broken selectors and schema mismatches.
- This breakthrough comes from experiments on 138 tasks, where the system consistently produced accurate results by using a structured approach involving JSON configurations and six-type collector taxonomy.
- The key innovation lies in shifting LLM output from free-form code to typed JSON structures, which ensures deterministic and verifiable execution paths.
- On 80 independently verified tasks, this method achieved zero execution-stage errors and the lowest average wall-clock time, prioritizing reusable and reliable data collection over initial quality.
- This advancement marks a significant step toward more dependable AI-driven web scraping tools, enabling developers to streamline repetitive data extraction processes with fewer manual interventions.
- The framework's ability to handle repeated tasks efficiently could pave the way for broader adoption in industries reliant on real-time data collection.
Terms in this brief
- JSON configurations
- A structured format for storing data in key-value pairs, used here to define web scraping tasks clearly and consistently.
- Collector taxonomy
- A classification system for different types of data collection methods, helping organize and standardize how web scrapers operate.
Read full story at arXiv CS.AI →
More briefs
ZLUDA Update Adds PhysX and Blender Support
ZLUDA has added two new major workloads: PhysX and Blender. This update also includes improved Windows support and better machine learning support. The new PhysX support means AMD GPU owners can play older games with higher frame rates and additional visual effects. For example, Mafia II can now run with maxed out settings and PhysX enabled on an AMD GPU. ZLUDA's updates will help gamers play classic games on newer hardware with better performance, and the project will continue to develop and improve.
AI Summaries Downplay Hotel Complaints
Tripadvisor's AI summaries of hotel reviews are downplaying serious complaints. A hotel being sued for food poisonings was described as spotless. A resort with guest complaints of sexual harassment by staff was praised for friendly service. Guests reported problems like raw chicken, flies on the buffet, and poor hygiene. The AI summaries are meant to help holidaymakers decide where to book. Tripadvisor will now look into the examples where reviews did not match the intended property.
BBVA Completes First AI-Initiated Transaction
BBVA completed a transaction initiated by an artificial intelligence agent on behalf of a cardholder. The payment was carried out using real card credentials and the systems of an active merchant. This transaction is important because it shows AI agents can securely complete purchases on behalf of cardholders within today's payments infrastructure. The transaction used Visa Payment Passkeys, a biometric authentication method that enables consumers to authorise online payments securely without passwords or SMS codes. 62% of surveyed consumers in Spain use AI to look for gift ideas, research products. The success of this transaction is a step forward for AI-enabled payments and BBVA will continue to explore how these new experiences can be delivered securely and at scale to cardholders. Next steps will be to further develop this technology to deliver seamless and reliable payment experiences.
AI Can Now Match Bookmakers' Accuracy in Predicting NFL Wins
Researchers have developed a new method for training AI to predict NFL game outcomes with unprecedented accuracy. By using a novel reward system based on past game data, the AI model achieved calibration levels matching professional betting markets without any human input or fine-tuning. This breakthrough addresses previous issues where AI predictions were less accurate due to noisy targets and corrupted reasoning chains. The innovation lies in a label-free reward mechanism that eliminates noise by estimating win rates from historical outcomes. By masking gradients during training, the AI maintains intact reasoning processes while avoiding corruption from policy gradients. This approach enables a 7B parameter model to reach betting market-level calibration through direct predictions alone. This development opens doors for more reliable probabilistic forecasting in various fields beyond sports analytics. Future research will focus on extending this method to other domains and improving generalization across different types of stochastic outcomes.
New Serverless Gateway Simplifies AI Agent Communication
A new serverless gateway has been developed to streamline communication between AI agents, reducing the complexity and operational burden for enterprises. As businesses deploy more AI agents across various teams and platforms, managing how these agents interact becomes increasingly challenging. Without a centralized system, each new agent requires multiple connections, leading to slower development cycles, security risks, and higher costs. This gateway introduces a single entry point that handles routing and access control, supporting agents running on AWS, other clouds, or hybrid environments. It uses the Agent-to-Agent (A2A) protocol to standardize communication. The solution includes three key layers: a management layer for agent discovery, a control layer for fine-grained permissions using JWT scopes, and an execution layer for routing requests through Amazon API Gateway and Lambda functions. This setup allows agents to connect behind a single domain with path-based routing, ensuring efficient and secure communication. By centralizing these processes, the gateway reduces the need for point-to-point connections, which would require 190 connections for just 20 agents. Looking ahead, this innovation could significantly accelerate AI adoption by simplifying agent integration and reducing operational overhead. Developers can now focus on building agent capabilities rather than managing connectivity. The use of streaming technologies like SSE ensures real-time responses, enhancing the functionality of AI-driven applications. As more enterprises adopt this gateway, it may become a standard for efficient AI agent communication.