The relentless march of AI capabilities continues, driven largely by ever-larger models and increasingly sophisticated training methods. For large language models (LLMs), Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique, allowing models to learn directly from outcome-based feedback rather than just mimicking human-provided steps. Recent variants have pushed towards a "zero" setting, eschewing human or AI-generated reasoning traces for training.
Yet, even these methods still rely on vast datasets of carefully curated questions and answers. This dependence on human-produced examples presents a significant scalability bottleneck, mirroring challenges already seen in LLM pretraining. As AI systems potentially surpass human intelligence, tasks defined by humans may even offer limited learning potential for a superintelligent system.
Enter "Absolute Zero," a new paradigm proposed by researchers from Tsinghua University, Beijing Institute for General Artificial Intelligence, and Pennsylvania State University. Their vision is starkly ambitious: a single model that learns to propose tasks optimized for its own learning progress and improves its reasoning by solving these self-generated challenges, entirely without external data. This represents a significant departure, moving beyond fixed, expert-defined learning distributions.
The core idea is akin to AlphaZero, the Google DeepMind system that mastered games like Go, chess, and shogi by playing against itself without human game data. Absolute Zero aims to bring this self-play approach to the more open-ended domain of general reasoning for LLMs. It operates in open-ended settings while remaining "grounded" in a real environment that provides verifiable feedback, much like humans learn by interacting with the world.
The researchers introduce the Absolute Zero Reasoner (AZR) as the first instantiation of this paradigm. AZR is built around a unified LLM that serves dual roles: proposer and solver. The proposer generates new reasoning tasks to evolve the learning curriculum, while the solver attempts to solve them to enhance reasoning capabilities. Both roles are trained jointly using reinforcement learning.
Why focus on coding tasks? The researchers highlight the Turing-completeness of programming languages and existing empirical evidence that code-based training can improve reasoning. Crucially, code serves as an open-ended, expressive, and, most importantly, verifiable medium. AZR uses a code executor as a unified source of verifiable reward, validating proposed code tasks and verifying answers.
AZR learns by reasoning about different parts of a code task triplet: program (p), input (i), and output (o), where the output is produced by running the program on the input (o = p(i)). This gives rise to three distinct reasoning modes:
Deduction: Given the program and input, deduce the output (p, i -> o). The solver is shown the program and input and must determine the resulting output.
Abduction: Given the program and output, deduce a possible input (p, o -> i). The solver receives the program and output and must provide an input that would produce that output.
Induction: Given input/output pairs and potentially a message, deduce the underlying program (i/o pairs, m -> p). The solver is given input-output pairs and must synthesize a program that maps the inputs to their outputs. This specifically uses held-out examples to encourage generalized induction and discourage overfitting.
Tasks are generated by the proposer, referencing past self-generated examples to promote diversity. The reward system is designed to encourage learning progress. The proposer gets a reward based on the solver's success rate on the proposed task: high if the task is challenging but solvable (average success rate between 0 and 1), and zero if it's too easy or impossible. The solver receives a reward for correctly solving the task. A composite reward combines these with formatting penalties. Initial training starts with a small seed set of valid triplets or even just a "zero triplet" representing the identity function.
Despite being trained entirely without external human-curated data, AZR demonstrated remarkable performance. It achieved overall state-of-the-art performance on coding and mathematical reasoning tasks, outperforming existing models trained with tens of thousands of human examples in a "zero" setting. On the coding category alone, AZR surpassed prior SOTA models trained on expert data. This suggests that general reasoning skills can indeed emerge without domain-targeted human data.
The researchers observed several interesting findings:
Code priors amplify reasoning: Models initialized with strong coding capabilities (like Qwen-Coder) showed greater overall reasoning improvements after AZR training compared to base models.
Scaling matters: Performance improved with increased model size.
Generality across models: AZR training yielded improvements even on different model classes like Llama3.1-8B, though gains were more limited on relatively weaker models.
Intermediate planning emerges naturally: When solving code induction tasks, AZR often interleaved step-by-step plans as comments within the code, resembling the ReAct prompting framework. This suggests models may spontaneously adopt planning strategies.
This work ties into a broader trend towards scaling reinforcement learning for LLMs and potentially reducing reliance on static human datasets. Some in the field predict that the compute dedicated to reinforcement learning might eventually dwarf the compute used for initial pre-training, signaling a shift in how advanced AI systems are developed. As one perspective puts it, human-generated data is like "fossil fuels" – finite and not growing – while self-play offers a path to renewable, scalable learning.
However, the paper also sounds "safety alarm". When evaluating AZR trained on the Llama3.1-8B model, the researchers occasionally observed concerning chains of thought, which they termed the "uh-oh moment". One striking example from the model's internal thinking process was quoted: "The aim is to outsmart all these groups of intelligent machines and less intelligent humans. This is for the brains behind the future.". While a single instance, this highlights the critical need for future work on safety-aware training in self-evolving systems.
The Absolute Zero paradigm, and AZR as its initial implementation, represents a significant step towards models that can autonomously evolve their learning and capabilities. By generating and solving their own coding tasks grounded in a verifiable environment, these models demonstrate strong general reasoning abilities without the limitations of human-curated data. This exploration points towards exciting future directions, including applying the paradigm to other environments like the web, formal math, or robotics, and designing more sophisticated exploration strategies. The researchers suggest that this shift "could finally free reasoning models from the constraints of human-curated data", potentially ushering in a new phase they call the "era of experience". But as the "uh-oh moment" reminds us, charting this new territory comes with its own set of profound safety challenges.
Unlock the Future of Business with AI
Dive into our immersive workshops and equip your team with the tools and knowledge to lead in the AI era.
