Brain's Complex Cost Functions Drive Efficient Learning Beyond AI

The brain’s profound learning efficiency, a mystery that has long eluded AI researchers, may hinge not on architecture or massive datasets, but on the intricate, evolutionarily-encoded reward functions that guide its learning. This conversation with Adam Marblestone reveals a critical gap in AI development: a neglect of these complex, often abstract cost functions. The implications are vast, suggesting that current AI, optimized for simple, mathematically tractable loss functions, may be fundamentally missing the nuanced drivers of true intelligence. Those who grasp the significance of these "steering subsystems" and their role in shaping learning will gain a significant advantage in developing more capable and aligned AI, moving beyond mere pattern recognition to a deeper form of understanding.

The Hidden Architecture of Desire: Why AI Needs a Better Reward System

The quest to replicate human-level intelligence in AI has largely focused on scaling up data, compute, and architectural complexity. Yet, as Adam Marblestone points out, the brain achieves remarkable learning efficiency with far less. The critical insight here is that the brain’s “secret sauce” might not be its neural architecture, but the sophisticated, deeply ingrained reward functions that dictate what it learns and how it learns it. Current AI, often trained on simplistic objectives like predicting the next token, may be missing the crucial, evolutionarily-shaped drivers that imbue biological intelligence with its remarkable adaptability and depth.

Marblestone, drawing on insights from neuroscience and the work of researchers like Steven Byrnes, posits that the brain operates with a "steering subsystem" -- a set of innate, evolutionarily-encoded reward functions and behaviors -- and a "learning subsystem," primarily the cortex, which models the world and learns to predict and interact with the steering subsystem. This framework suggests that evolution has encoded not just basic survival instincts, but also the curriculum for learning, through these complex reward signals.

"Evolution could encode the knowledge of of the learning curriculum so so in the machine learning framework maybe we can come back and we can talk about yeah where do the loss functions of the brain come from can that can loss different loss functions lead to different efficiency of learning"

This is a radical departure from the current AI paradigm, which often relies on mathematically simple, human-defined loss functions. The brain, Marblestone suggests, has a rich, multi-faceted reward system that guides learning in a way that current AI cannot replicate. This system allows the brain to generalize and adapt to novel situations, even those never explicitly encountered during its evolutionary history. For instance, the innate fear of spiders, encoded through the steering subsystem, can generalize to the word "spider" or the concept of a spider, even without direct, supervised learning on those specific instances. This generalization capability, driven by the reward function, is a key differentiator.

The implications for AI are profound. If current models are primarily sophisticated prediction engines optimizing for narrow objectives, they may lack the foundational drivers for true understanding and general intelligence. The challenge, then, is to move beyond simple loss functions and explore how to imbue AI with more complex, perhaps even evolutionarily-inspired, reward structures.

The Genome's Compact Wisdom: Encoding Desire Without Explicit Instruction

A central puzzle Marblestone explores is how the genome, with its limited information capacity, can encode such complex desires and reward functions. Evolution, he argues, doesn't explicitly encode knowledge of specific future scenarios (like Yann LeCun listening to a podcast). Instead, it encodes heuristics and general principles that allow the learning subsystem to wire learned features of the world to innate reward functions.

"Evolution has to make sure that that those neurons whatever the yann lecun being upset with me neurons get properly wired up to the shame response or this part of the reward function"

This suggests that the genome provides a foundational blueprint for the steering subsystem, which then guides the learning subsystem. The cortex, as the learning subsystem, builds a world model, and its success is partly judged by how well it predicts and aligns with the outputs of the steering subsystem. This creates a feedback loop where learned representations are mapped to innate reward functions, enabling generalization and adaptation. The distinct cell types found in the steering subsystem compared to the cortex further support this idea of specialized, innately wired circuits for reward and behavior.

Omnidirectional Inference: The Brain's Flexible Prediction Engine

Marblestone highlights the concept of "omnidirectional inference" as a potential key difference between brain and AI. While LLMs are primarily trained to predict the next token, the cortex, he suggests, may be capable of predicting any subset of its inputs given any other subset. This allows for a much more flexible and general form of prediction and inference, akin to probabilistic AI or energy-based models championed by researchers like Yann LeCun.

"Whereas an llm is just you see everything in the context window and then it it computes a very particular conditional probability which is given all the last thousands of things what is the very probability for all the all the next token"

This omnidirectional capability could explain the brain's efficiency and its ability to learn from limited data. It’s not just about predicting what comes next, but about understanding the underlying causal relationships and being able to infer missing information from diverse perspectives. This is a stark contrast to the unidirectional, next-token prediction of many current LLMs, suggesting that true intelligence requires a more holistic, probabilistic approach to understanding the world.

The Value of Primitive Reinforcement Learning

The discussion also touches upon reinforcement learning (RL) in both AI and the brain. Marblestone expresses surprise that current LLMs largely eschew value functions, a core component of more sophisticated RL algorithms like Q-learning, which were inspired by neuroscience. He suggests that parts of the brain, like the basal ganglia, likely employ simpler forms of model-free RL, while the cortex, with its world model, can engage in more complex model-based RL, predicting rewards based on learned plans and circumstances.

The idea of "amortized inference" -- where complex inferential computations are baked into the model during training rather than performed at test time -- is also explored. This parallels how LLMs, after extensive training, can perform complex tasks in a single pass. However, the brain’s energy efficiency and its ability to handle unstructured sparsity suggest that it may have found ways to combine both amortized and non-amortized inference, a balance that AI is still striving to achieve.

Actionable Insights for the AI Frontier

The conversation with Adam Marblestone offers a compelling framework for re-evaluating AI development. The emphasis on reward functions, innate drives, and flexible inference points towards a future where AI development might shift from optimizing architectures to engineering more sophisticated, biologically-inspired learning signals.

Key Action Items

• Prioritize Reward Function Research: Investigate and develop AI architectures that can learn from and implement complex, multi-objective reward functions, moving beyond simple next-token prediction or cross-entropy loss. (Immediate Action)
• Explore "Steering Subsystem" Analogues: Research how to imbue AI with innate priors or "drives" that guide learning towards more generalizable and robust understanding, potentially inspired by evolutionary heuristics. (Ongoing Investment)
• Develop Omnidirectional Inference Models: Shift focus from unidirectional prediction to AI systems capable of flexible, probabilistic inference across multiple variables and modalities. (12-18 Month Investment)
• Integrate Model-Based and Model-Free RL: Explore hybrid architectures that combine the efficiency of model-free RL with the predictive power of model-based RL, mirroring potential brain mechanisms. (Immediate Action)
• Invest in Neuroscience-Inspired AI Architectures: Support research into brain-inspired computational principles, such as energy-based models or architectures that support more general prediction. (Ongoing Investment)
• Quantify "Generalization" Drivers: Develop metrics and methodologies to explicitly measure and improve the generalization capabilities of AI, particularly those stemming from reward function design. (Immediate Action)
• Foster Cross-Disciplinary Collaboration: Encourage deeper collaboration between AI researchers and neuroscientists to bridge the gap between current AI capabilities and the brain's learning mechanisms. (Immediate Action)