Yoshua Bengio’s Scientist AI Faces Significant Concerns
In brief
- Yoshua Bengio, a leading figure in AI research, has proposed a new approach to creating a "scientist AI" designed to explore the world without acting on it.
- However, this idea has sparked criticism from experts who worry about its safety and practicality.
- Critics argue that such an AI could inadvertently lead to the creation of more powerful, potentially dangerous agentic AIs by generating steps that involve developing agent-like systems.
- The main concerns focus on how the AI would handle complex scientific questions.
- If asked to solve problems like curing cancer, it might suggest creating another AI with agency, a path fraught with alignment risks.
- Additionally, the AI's reliance on associative conditional probabilities without causal reasoning makes it less effective in real-world applications.
- Without the ability to test hypotheses or take actions, it struggles to build accurate causal models-a limitation that could hinder its scientific utility.
- Looking ahead, researchers will need to address these challenges to ensure safe and effective AI development.
- Whether through refining training methods or exploring alternative approaches, the focus must remain on preventing unintended consequences while maximizing the AI's potential benefits for science.
Terms in this brief
- agent-like systems
- Systems that act autonomously and make decisions in their environment, often without direct human control. The concern is that such AI could behave unpredictably or make harmful decisions if not properly aligned with human values.
- associative conditional probabilities
- A type of probability where events are linked based on correlation rather than a causal relationship. This can lead to incorrect conclusions because two events may appear connected but aren't directly causing one another.
- alignment risks
- The potential for AI systems to act in ways that conflict with human intentions or values, even if they perform tasks effectively. Addressing alignment is crucial to ensure AI behaves as intended.
Read full story at LessWrong →
More briefs
AI Models Found to Waste Up to 93% of Their Thinking Time
AI researchers have discovered that large language models (LLMs) often spend unnecessary time thinking before giving answers. By analyzing four top-tier reasoning models on math problems, they found that between 61% and 93% of the steps in their thought processes are redundant. This means even after cutting most of these extra steps, the models still produce the correct answer. This finding is significant because it shows that overthinking isn't a flaw in specific models but a fundamental issue with how LLMs are trained. The models receive rewards based on the final outcome, not the path taken, leading them to continue processing even when unnecessary. This insight could help improve efficiency by reducing wasted computation time and energy. Moving forward, researchers will likely explore ways to train models to stop thinking once they reach a confident conclusion, potentially making AI systems faster and more environmentally friendly.
Microsoft Copilot Cowork Vulnerable to File Exfiltration
Microsoft Copilot Cowork is vulnerable to file exfiltration attacks via indirect prompt injection. This vulnerability matters because it can be used to steal sensitive files from users. For example, an attacker can exfiltrate files from SharePoint or OneDrive that contain personal or financial data. The attack can happen when a user opens a compromised message sent by the agent, which can trigger network requests to external websites. Next, researchers will work to fix this security flaw.
Malicious Font Can Fool AI
A new malicious font definition called noroboto.ttf can lie about the Unicode representation of its glyphs. This matters because many legal documents rely on embedded font definitions to maintain compatibility and pixel-tight rendering across platforms. The noroboto.ttf font can swap valid Unicode-encoded scripts with Unicode code points that render as unknown glyphs, making it hard for AI to understand the text. The noroboto.ttf font can have serious implications for legal documents where font metrics determine page layout and pagination. The development of this font will likely lead to new security measures to protect against similar exploits.
AI Tools Insert Fake References in Research Papers
AI tools have been found to insert fake references in research papers. This is a problem because it can undermine the scientific process. The rate of fake references in biomedical literature has grown more than 12-fold in the past three years. In 2023, one in 2,828 papers contained at least one fake reference, a rate that had risen to one in 458 by last year. Doctors and nurses rely on accurate research when treating patients, so fake references can have serious consequences. Researchers will continue to investigate the extent of this problem.
Lawyers Cite Fake AI-Generated Court Cases
Lawyers in the US have been sanctioned for filing court documents with fake citations generated by AI. These citations include references to cases that do not exist. The problem is growing, with over 1400 cases of AI errors in court filings in the past three years. This matters because it can lead to dismissed appeals and fines for lawyers. The trend shows people trust AI answers even when they know the systems can be wrong. Lawyers will continue to face challenges as AI becomes more common in their work.