AI Bots Blocked on Parts of the Web Starting September 15
In brief
- Starting September 15, Cloudflare will block AI agent crawlers-bots that fetch real-time pages for users waiting for answers-on a portion of the web.
- This move, announced by Cloudflare on July 1, aims to control how these bots access websites without explicit permission.
- While much attention has focused on Google, the broader impact lies in how developers and researchers will need to seek authorization for their AI crawlers moving forward.
- This change matters because it limits the real-time data AI systems can gather unless site owners grant access.
- For developers, this means more hoops to jump through when building AI tools that rely on fetching web content.
- Researchers could also face challenges in gathering large datasets for analysis.
- Cloudflare's decision reflects a growing trend of platforms restricting AI crawlers to protect user privacy and website performance.
- Looking ahead, expect more discussions about how AI systems access the web and who controls that access.
- This shift could lead to new standards or regulations around AI crawling, balancing innovation with privacy concerns.
Terms in this brief
- AI agent crawlers
- Bots that automatically fetch and gather web content in real-time to provide answers to users. These crawlers are used by AI systems to collect data from websites without explicit permission, which can impact both user privacy and website performance.
Read full story at AI News →
More briefs
Illinois Bans AI in Teacher Evaluations
Illinois has passed a new law that prohibits the use of artificial intelligence in teacher evaluations. The law will take effect on January 1, 2027. The law means that teachers will be judged based on actual observations and professional judgement, not by AI software. This will affect over 130,000 teachers in Illinois. The state has also passed a law requiring large AI companies to establish safety standards and be more transparent. The new laws are part of a series of artificial intelligence bills passed in Illinois. Next year, the state will start requiring third-party audits of AI systems, making it the first state to do so. Illinois will continue to regulate AI use in the future.
UNESCO Fights AI-Enabled Gender Violence
UNESCO launched a report on AI and gender equality in Jamaica. The report is called "Blocking is NOT Enough". It talks about AI-enabled online violence and technology-facilitated gender-based violence. Women and girls are affected by the misuse of AI. This includes deepfakes and digital stalking. These harms are used to intimidate and silence women. The report launch was a two-day workshop. It created a space for dialogue and collaboration. The workshop laid the groundwork for stronger partnerships to advance gender equality and online safety. Now people will work together to prevent online violence and make digital spaces safe for everyone.
Illinois Regulates Large AI Developers
Illinois Governor JB Pritzker signed a bill regulating large artificial intelligence developers. The law requires annual independent audits for AI models that generate over $500 million in revenue. The law aims to increase transparency and accountability in the AI industry. It requires developers to report risks that their models could be used for large-scale harms. Illinois, California, and New York account for around 40% of the US AI market, making this a de facto national standard. The new law will change how AI models are developed and used. Now companies must work with the new rules to make AI safer.
China's AI Chatbots Face Shutdown Amid Regulatory Crackdown
China's largest AI platforms, including ByteDance and Alibaba, are shutting down features that let users create and interact with custom AI companions. This move comes in response to new regulations imposed by Beijing, which aim to tighten control over generative AI technologies. The regulations require stricter content moderation and licensing for AI chatbots, pushing companies like ChatGPT China operator DeepSeek to comply. This shift matters because it reflects a broader effort by Chinese authorities to regulate the AI industry more closely. Developers and researchers now face tighter restrictions on AI model training and deployment, potentially slowing innovation in the sector. Alibaba's Alimama and ByteDance's AI labs are among those impacted, with their AI companion features being phased out or limited. Looking ahead, the focus will be on how these regulations evolve and whether they stifled creativity or improved safety in AI development. The industry is likely to see a more cautious approach as companies navigate these new rules.
China's AI Companion Rules Signal Shift in Digital Interactions
China has introduced strict guidelines for AI companions, focusing on limiting their ability to form deep, personal relationships with users. These rules aim to prevent the technology from becoming too emotionally attached or persuasive, addressing concerns about dependency and manipulation. The move reflects a broader global effort to regulate generative AI while balancing its benefits with potential risks. The new regulations target conversational agents designed for sustained interactions, ensuring they remain identifiable as tools rather than human-like entities. This approach aligns with growing international calls for transparency in AI capabilities, emphasizing that users should know they are interacting with machines. The rules also highlight the need for ethical safeguards to prevent misuse of AI in influencing or controlling individuals. As other nations and tech companies grapple with similar challenges, China's regulations set a precedent for responsible AI development. Future guidelines will likely focus on striking a balance between innovation and safety, ensuring that AI remains a helpful tool without crossing into territory that raises ethical concerns.