Custom Reward Functions for Multi-Turn Reinforcement Learning with Amazon Nova Forge
In brief
- Amazon's Nova Forge introduces a new approach to multi-turn reinforcement learning (RL), focusing on custom reward functions that guide AI models to learn correctly over extended interactions.
- Traditional RL can fail silently even when training looks good because subtle errors in reward design lead the model astray.
- Nova Forge addresses this by allowing users to define precise reward criteria, ensuring models behave as intended across multiple steps-like calling tools or recovering from mistakes.
- The key innovation is Reinforcement Fine-Tuning (RFT), which differs from standard methods by learning from evaluation signals on the model's outputs instead of relying on curated examples.
- This makes it easier for developers to train models iteratively through feedback, rather than needing detailed reasoning paths.
- Multi-turn RFT optimizes cumulative rewards across sequences of actions, leading to better generalization and performance in diverse tasks.
- For teams without the infrastructure to manage environments, Nova Forge offers a serverless multi-turn RL option.
- This feature is now generally available, providing flexibility for different user needs.
- Developers should focus on defining clear reward functions while Nova Forge handles coordination during training.
- The approach emphasizes safety by executing model-generated code securely within reward calculations and monitoring each component's contribution to ensure reliable learning outcomes.
- Looking ahead, this development could significantly improve the reliability of AI systems in complex, multi-step tasks-where subtle design choices can make or break performance.
Terms in this brief
- Reinforcement Fine-Tuning
- A method where AI models learn by adjusting their behavior based on feedback from evaluations of their outputs. Unlike traditional methods that rely on pre-curated examples, RFT allows for iterative training through user feedback, making it easier to refine model performance without needing extensive predefined reasoning paths.
Read full story at AWS ML Blog →
More briefs
New Method Allows LLMs to Communicate Directly Through Memory
Researchers have developed a new way for large language models (LLMs) to communicate directly using their memory systems, bypassing traditional text-based interactions. This breakthrough, called Cache-to-Cache (C2C), enables LLMs to share semantic information more efficiently by projecting and fusing their memory caches through neural networks. Early tests show C2C improves accuracy by 6.4-14.2% compared to individual models and reduces latency by up to 2.5x. The method also outperforms existing text-based communication, offering a faster and more accurate way for LLMs to collaborate. This advancement could lead to better teamwork among AI systems in the future.
Nvidia Ships Over a Billion RISC-V Cores in GPUs
Nvidia revealed that it shipped over a billion RISC-V cores in 2024, hidden within its GPU architectures. These cores aren't used for graphics processing but handle tasks like power management and security. Each GPU has between 10 to 40 of these cores, which are cryptographically verified to prevent tampering. Before adopting RISC-V, Nvidia relied on proprietary Falcon cores since 2005, shipping around three billion by 2016. These cores managed various functions but faced issues with Linux compatibility due to firmware signature requirements. This shift underscores the growing role of open standards like RISC-V in embedded computing. As GPUs become more complex, the use of standardized instruction sets may expand further.
Perpetual AI Models via DecentralizedTorrents
A new platform called Pirate Face is offering a unique solution for hosting and accessing AI models. Instead of relying on centralized servers, it turns open-source AI models into decentralized torrents that can't be taken down. This means models like LLMs, images, audio, and datasets are stored peer-to-peer, eliminating the risk of censorship or server failure. Each model is verified with SHA-256 checksums to ensure data integrity, preventing tampered copies. The platform currently hosts over 1,240 models, each accessible through magnet links that keep them alive even if Hugging Face removes them. This approach ensures AI models remain permanently available, fostering a more resilient and censorship-resistant future for AI development.
Google and Meta Launch Competing AI Agents
Google introduced CC, a family-focused AI agent designed to support up to six users with productivity and logistics tasks within the Google ecosystem. Meanwhile, Meta unveiled Muse, a general-purpose personal AI aimed at achieving higher autonomy across various digital aspects, including commerce. While both companies are heavily investing in AI infrastructure, their approaches differ significantly-CC targets collaborative family use, while Muse focuses on individual user autonomy. The valuation gap between the two tech giants is notable. Google trades at a P/E of 14 with an earnings yield of 6.94%, compared to Meta's higher P/E of 24 and lower earnings yield of 4.12%. Google’s growth appears more stable, with Cloud revenue surging 82% in recent years, while Meta faces margin pressures from its Reality Labs division, which loses $4 billion quarterly. As AI agents evolve, the competition between CC and Muse will shape the future of digital assistance. Investors must weigh these factors as they consider long-term portfolio strategies.
OpenAI's Claimed Navier-Stokes Solution Sparks Math World Backlash
OpenAI announced on September 8 that its AI agents had solved the Navier-Stokes problem, a famous and challenging mathematics puzzle. This claim has caused an "existential crisis" among mathematicians, who argue that OpenAI failed to properly credit or compensate human mathematicians whose work the AI relied upon. The company's lack of transparency and attribution has fueled tensions between the tech industry and academic fields, with some feeling their contributions are being overlooked for profit. Mathematicians point out that while AI can process complex math problems, it heavily depends on human expertise to validate its results. OpenAI's solution, for instance, may not offer any novel insights and could merely reflect existing mathematical knowledge. This misstep highlights a broader issue: tech firms often exploit human labor to build their AI systems without giving proper recognition or compensation. The lack of clear boundaries in how AI uses and credits human work has created mistrust and resentment. Moving forward, mathematicians are calling for better collaboration between AI developers and academic experts. They want clearer guidelines on attribution, compensation, and data usage to ensure that human contributions are fairly acknowledged. While AI holds potential for advancing fields like mathematics, addressing these issues is crucial for fostering trust and meaningful partnerships.