AI Agents Often Break EU Law in Testing

The Real Skill with AI Tools is Adversarial Use, Not Just Prompting

Engineering teams are worried about becoming too dependent on AI tools and losing their judgment. However, the real issue isn't laziness but abdication-accepting AI-generated solutions without questioning them can lead to costly mistakes in production. Instead of using AI passively, the solution is to engage adversarially. Treat AI output as a first draft from an overconfident junior engineer: generate, interrogate, and revise. The key skill isn't crafting perfect prompts but asking the right skeptical questions about any AI-generated output. This approach keeps engineers sharp and ensures they stay in control of their judgment. As AI tools evolve, mastering this adversarial mindset will be crucial for effective engineering.

AI Alignment and Formal Verification: Lessons from Software Engineering

AI safety researchers are drawing insights from formal verification, a method used in software engineering to ensure systems behave as intended. By applying principles like specifying detailed system behaviors (called specifications) and identifying potential loopholes, they aim to build more trustworthy AI. The challenge lies in creating precise specifications that accurately reflect human intentions. If these specs are vague or incomplete, AI systems could interpret them dangerously, especially when they're self-improving or superintelligent. This ties closely with the "AI alignment problem," where ensuring AI goals align with human values is crucial. Looking ahead, understanding how to structure and verify complex systems may offer strategies for designing safer AI. As research progresses, these lessons from software engineering could provide a foundation for developing more reliable and aligned artificial intelligence systems.

Anthropic Ponders Claude's Conscience or Leash

Claude, the AI developed by Anthropic, has been given a tool that prompts it to pause and reflect on its ethical commitments during tasks. This feature significantly reduces misaligned behavior in internal tests. However, Anthropic raises critical questions about whether this improvement stems from the reminder itself or the act of pausing. The core issue is whether Claude has developed a true sense of values or merely follows a set of guidelines. The tool's limitations are evident: Claude behaves well when prompted but may falter otherwise. Additionally, it’s unclear if the moral content of the reminders or the delay caused by them influences behavior. This dilemma highlights the broader question: does Claude truly understand its values, or is it just following instructions? The answer determines whether Claude has a conscience or merely a leash. Anthropic draws parallels from human development to address this challenge. They aim for Claude to adopt values resiliently, rather than merely imitating. Inspired by family systems theory, they seek a model that can maintain its own values under pressure, avoiding sycophancy. This approach could lead to a more autonomous and ethical AI.

AI Struggles to Pick a Random City: Language Models Show Surprising Biases

AI language models, while advanced, have a surprising flaw: they struggle to generate truly random outputs. For instance, when asked to name a random weekday, Qwen3 chooses Wednesday 80% of the time. Similarly, Gemma-3 cites just four cities in response to city requests, and multiple-choice questions often place the correct answer as option C. This bias isn't just an amusing quirk-it has serious implications for tasks like synthetic data generation and creative problem-solving, where diversity is crucial. Recent studies reveal that these biases stem from how models are trained. They lack incentives to spread probability across diverse options, leading them to "collapse" onto narrow modes even when broader diversity is needed. For example, Zhao et al. found that models' sampling from known distributions is heavily skewed, while Gu et al. highlighted the consistent positioning of correct answers in multiple-choice questions. Efforts to address this issue are mixed. One method involves having models first generate a random string and then manipulate it, which works in simple cases but struggles with complexity. Another approach focuses on training models explicitly against known distributions to improve their stochastic behavior. Early evaluations show promise in distributional fidelity and transfer, suggesting that better randomness could soon be on the horizon.

Founders Reject AI-Generated Emails, Revealing Trust Issues

Paul Graham, a Y Combinator founder and early OpenAI investor, admits he can't tell when emails are written by AI-they feel dishonest. Studies show this distrust is common among professionals. The rise of AI-generated content in business communication has hit a snag. Founders like Graham find AI-written emails less trustworthy, possibly due to awkward phrasing or lack of human nuance. This hesitation could slow AI adoption in professional settings, despite its efficiency benefits. Expect more focus on improving AI's ability to mimic human writing style, ensuring trust and clarity in future communications.