Human Judgment and Reinforcement Learning Drive AI Progress

Recursive self-improvement does not follow a straight line. It moves through a series of difficult, sudden shifts. While current scaling laws suggest a steady path toward general intelligence, this discussion shows that the real bottleneck is not compute power. It is the human side of defining objectives. Organizations gain an advantage by realizing that intelligence is not one single ability. It is a collection of meta-skills like persistence, triage, and long-term judgment. Those who use AI as a worker today while building systems to verify and distill its output will see compounding gains over the next three years. This represents a fundamental change in how research and production work, favoring those who can bridge the gap between simulated training and real-world results.

The Hidden Dynamics of AI Progress

The Sim-to-Real Trap and the Bottleneck of Taste

Conventional wisdom suggests that scaling current architectures will lead to recursive self-improvement. However, the panelists argue that the gap between simulation and reality remains a persistent, hidden barrier. We are currently limited by the human need to specify goals. Even if a model writes code faster than a person, it lacks the taste to judge which systems will remain maintainable and performant over time.

Humans have a lot of advantages over models now and each time a new model comes out it will sort of catch up in some of these areas but you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment or the models cannot check themselves well enough.

-- John Schulman

This suggests that the singularity is not a single event, but a series of phase transitions. Each breakthrough, from next-token prediction to reinforcement learning, has kept the progress curve looking linear. This masks the fact that we constantly hit diminishing returns that require new paradigms to overcome.

The Efficiency of RL vs. The Noise of Data

A key insight is that reinforcement learning is not just more training. It is a massive increase in the signal-to-noise ratio. In supervised fine-tuning, a model mimics the exact reasoning of its teacher, absorbing noise and irrelevant habits. Reinforcement learning focuses the model on the single bit of information that leads to a correct outcome.

For RL you only get the one bit, and that signal is not drowned out in the noise of all the other bits the model has, and so that is what really is a super dramatic increase in signal-to-noise ratio as you are doing training.

-- Beren Millidge

This explains why small, targeted updates can change a model behavior. Developers should stop trying to feed models every possible data point and start focusing on high-signal, verifiable environments where the model learns how to succeed rather than just copying a specific dataset.

Why Consolidation Is Not Guaranteed

While current trends favor large labs, distillation is the primary force countering this. If a behavior can be learned through reinforcement learning, it can be distilled into a smaller, more efficient model. The advantage of big labs is not the models themselves, but their ability to build high-difficulty, realistic environments that force these capabilities to emerge. Competitors who can synthesize these distributions can close the gap, even without access to frontier model weights.

Key Action Items

• Audit your verification bottleneck: As agent-generated code becomes standard, your risk shifts from writing code to verifying it. Over the next quarter, invest in automated testing and verification tools to ensure agent output does not create technical debt.
• Shift from data to environments: Stop treating internal data as a static training set. Over the next 6-12 months, focus on converting your most difficult, repeatable workflows into environments where agents can be trained via reinforcement learning.
• Prioritize taste in your stack: Recognize that models will struggle with long-horizon judgment for some time. Dedicate engineering time to defining the constitution or model specs for your domain. This is the final, non-automatable layer of alignment.
• Distill, do not just prompt: If you rely on frontier models, do not just use them via API. Use their outputs to generate synthetic data and distill those behaviors into smaller, specialized models that run locally. This reduces latency and dependency over the next 12-18 months.
• Prepare for a 10x shift in research velocity: The panelists suggest that AI will provide a 10x productivity boost for researchers within two years. Start planning now for how your organization will handle a 10x increase in the volume of experiments and research output.