Category Theory: A Principled Framework for AI Computation

The current state of Artificial Intelligence, particularly Large Language Models (LLMs), is akin to a brilliant but flawed alchemist. While these models can produce astonishing results, they fundamentally lack true understanding, struggling with basic logical operations like addition or consistent adherence to physical laws. This conversation reveals that the "magic" we see is pattern recognition, not genuine reasoning. The hidden consequence is that relying on these models for complex tasks is precarious, as they can be easily tripped up by deviations from learned patterns. This analysis is crucial for researchers, engineers, and anyone building or deploying AI systems, offering a glimpse into a future where AI is grounded in rigorous mathematical principles, moving beyond trial-and-error to principled engineering. Understanding this shift provides a significant advantage in developing more robust and reliable AI.

The Illusion of Understanding: Why LLMs Can't Truly Add

The most striking revelation from this discussion is the fundamental disconnect between how LLMs appear to "understand" and how they actually process information. While they can mimic mathematical operations and even physics, their capabilities are rooted in pattern matching, not in the internal machinery of computation. Andrew Dudzik highlights this with the example of addition: LLMs can often produce correct sums because they recognize patterns in number sequences, but they fail when a subtle change disrupts that pattern. This isn't a minor flaw; it means that even sophisticated models can be "tripped up" by a single digit change, failing to perform a basic carry operation.

"language models cannot do addition not really i keep seeing claims that they can and every time i see this claim i go again to chatgpt and so on and check and they can't and uh what they can do is learn patterns which work a lot of the time but you can always trip them up by doing something like okay so if you ask chatgpt what is a bunch of eights plus a bunch of ones with a two at the end it will get the correct answer because it will recognize the trick it'll say ah that's just one and a bunch of zeros it'll know that you're trying to trick it but if now you change one of the eights to a seven now it has to actually know what it's doing it has to actually sort of walk up hit the seven and stop sort of propagating zeros and it simply fails"

This failure points to a critical limitation: current LLMs lack the intrinsic "machinery" for robust algorithmic reasoning. The implication is that relying on them for tasks requiring precise calculation or logical deduction is inherently risky. The conversation suggests that while tool use (like hooking an LLM up to a calculator) can augment capabilities, it doesn't solve the underlying architectural problem. The true efficiency and stability gains will come from models that internalize these computational abilities. This is where the "alchemy" phase of deep learning, as described by the speakers, becomes apparent -- powerful results are achieved through empirical exploration, but without a unifying theoretical framework to guide development.

Beyond Group Theory: Category Theory as the "Periodic Table" for AI

The discussion pivots to the limitations of current architectural paradigms, particularly Geometric Deep Learning (GDL), which heavily relies on group theory to encode symmetries. Petar Veličković points out that while GDL is powerful for handling spatial regularities and symmetries (like translation invariance in images or permutation invariance in graphs), it falls short when dealing with generic computation. Many algorithms, by their nature, involve information destruction or non-invertible operations, which don't neatly fit into the framework of group theory.

This is where Category Theory emerges as a potential solution. Taco Cohen frames it as "algebra with colors," a way to compose systems that have matching "types" or "colors," much like how matrix multiplication requires inner dimensions to match. This concept of "partial compositionality" is key. It allows for more flexible composition of operations, acknowledging that not all computations can be seamlessly chained together.

"the point is a situation where we want to be able to compose things -- to to hook them up together -- but we can't always do it that's basically what categories are designed to cover and i think the the matrix example illustrates they're not so mysterious it's just when you want to be talking about for example many different sized vector spaces at once as you often do in neural networks because you have sort of hybrid shapes of with you know dimensions of of different sizes and so on you end up you end up wanting something where you take this sort of partial compositionality into account"

Category theory offers a systematic, Lego-like approach to building complex systems. Instead of stumbling upon architectures through trial and error, it provides a principled framework for constructing neural networks that respect not just symmetries, but also more general structures and structure-preserving maps. This shift from "alchemy" to rigorous engineering is precisely what's needed for advancing towards Artificial General Intelligence (AGI). The implication is that by imbuing models with categorical priors, we can move beyond mere prediction to genuine understanding and reasoning, making AI capable of tackling complex scientific problems.

The Synthetic Approach: Building Knowledge from Structure, Not Substance

Paul Lessard introduces a crucial philosophical distinction: moving from "Analytic" mathematics to "Synthetic" mathematics. Analytic math focuses on what things are made of, breaking them down into fundamental components. Synthetic math, on the other hand, abstracts away the underlying substance and focuses purely on the principles and rules of inference that govern relationships between entities. This is where category theory truly shines. It's a form of "structuralist mathematics," concerned with the relationships and transformations between objects, rather than the objects themselves.

This synthetic, structure-first approach is vital for AI. Instead of trying to model every physical detail (which can be computationally intractable and prone to error), category theory allows us to encode the essential structures and rules of computation. This is exemplified by the concept of "weight tying," where identical parameters are used across different parts of a computation. While traditionally an ad hoc implementation detail, category theory provides a formal framework to understand when and why weight tying is valid, and crucially, how it preserves desired structures.

"we shouldn't just think of them as categories they're not just maps but they have this higher we can think of them certainly like this but that does not encode a lot of the interesting things we want to have about them and this is i think the idea of higher categories where you start modeling something with purely categories and you realize ah well all along i have been forgetting about this other important thing so you start putting more stuff into your theory well still trying to make it consistent and the particular way we encode these higher morphisms or what they what we use them for i think the most important thing is weight sharing"

The idea is to build AI systems that understand the "syntax" of computation -- the rules and structures -- rather than just its "semantics" or underlying implementation details. This focus on structure-preserving maps, or homomorphisms between algebras, is the core proposal of categorical deep learning. It promises to unify different perspectives on AI, from geometric deep learning to algorithmic reasoning, offering a path towards more robust, interpretable, and generalizable AI systems. The advantage here lies in building AI that is fundamentally more aligned with how we reason and how the universe operates, rather than relying on approximations that can easily break.

Key Action Items

• Immediate Action (Next 1-3 Months):

  • Educate Yourself on Category Theory Fundamentals: Begin exploring introductory resources on category theory, focusing on concepts like objects, morphisms, functors, and natural transformations. This foundational understanding is critical.
  • Identify "Carry" Problems in Your Systems: Actively look for areas in your current AI/ML systems where operations akin to carrying or information propagation are critical but potentially brittle. This might involve numerical computations, state transitions, or complex data processing pipelines.
  • Experiment with Tool Use for Algorithmic Tasks: While not a long-term solution, continue to explore and refine the integration of LLMs with external tools (like calculators or symbolic engines) to understand the practical benefits and limitations of this approach.

• Near-Term Investment (Next 3-9 Months):

  • Explore Geometric Deep Learning Concepts: Familiarize yourself with the principles of Geometric Deep Learning, particularly equivariance and invariance, to understand the landscape of symmetry-aware AI.
  • Investigate "Synthetic" vs. "Analytic" Approaches: Consider how your current AI development leans towards analytic (breaking down into components) versus synthetic (focusing on structural rules) approaches. Identify opportunities to adopt more synthetic, structure-preserving methods.
  • Prototype with Compositional Frameworks: If possible, experiment with libraries or frameworks that offer more explicit compositional capabilities, even if they are not fully categorical. The goal is to gain practical experience in managing partial compositionality.

• Long-Term Investment (9-18+ Months):

  • Explore Categorical Deep Learning Research: Stay abreast of advancements in categorical deep learning, including papers and implementations that demonstrate its application to specific problems like weight tying or algorithmic reasoning.
  • Develop Internal Expertise: Foster or acquire expertise in category theory and its application to machine learning within your team. This is a significant undertaking but offers a substantial competitive advantage in building next-generation AI.
  • Rethink Model Architectures for Internalization: Begin planning for future AI architectures that aim to internalize core computational and reasoning capabilities, rather than relying solely on external tools or pattern matching. This may involve exploring nascent categorical architectures. This investment creates a durable moat as it requires significant theoretical and practical groundwork that few are willing to undertake.