AI's Missing Dimension: World Models for 3D Spatial Intelligence

Key Quotes

"the space the 3d space the space out there the space in your mind's eye the spatial intelligence that enables people to do so many things that's beyond language is a critical part of intelligence"

Fei-Fei Li argues that spatial intelligence, which allows for understanding and interacting with the 3D world, is a fundamental aspect of intelligence that extends beyond language capabilities. This highlights her core belief that current AI is missing a crucial component by focusing primarily on linguistic understanding.

"but what if we're missing something more fundamental not words but space the physical world we move through and shape my guests today think we are"

This statement introduces the central thesis of the podcast episode, suggesting that the dominant focus on language models (LLMs) in AI might be overlooking a more essential element: the understanding and interaction with the physical, spatial world. It sets up the discussion around the importance of "world models."

"feifei really singularly brought in data to the equation which now we're recognizing is actually probably the bigger problem the more interesting one and so she truly is the godmother of ai as everybody calls her"

Martin Casado highlights Fei-Fei Li's pivotal contribution to AI by emphasizing her role in bringing data to the forefront of machine learning research. He suggests that the significance of data in AI development is now widely acknowledged, solidifying her reputation as a foundational figure in the field.

"i was also particularly looking for an intellectual partner because what we are doing at world labs is very deep tech we are trying to do something no one else has done we know with a lot of conviction it will change the world literally but i need someone who is a computer scientist who's a student of ai understand product market go to market go to market and just can be on the phone or in person with me every moment of the day as an intellectual partner"

Fei-Fei Li explains her criteria for selecting Martin Casado as an investor, emphasizing the need for an intellectual partner due to the deeply technical and novel nature of World Labs' mission. She sought someone with a strong computer science and AI background who could provide both strategic insight and constant collaboration.

"and so feifei if i'm in the end of this table all these people talking about it and feifei leans over to me she's like you know what we're missing i said what are we missing she said we're missing a world model and i'm like yes and it fell into place then because i'd been like thinking about stuff at a high level but as she does she just kind of perfectly articulated this"

Martin Casado recounts the moment of alignment with Fei-Fei Li regarding the concept of "world models," contrasting it with the prevailing LLM discussions. He describes how Li's precise articulation of this missing piece resonated with his own developing intuition, leading to a shared vision.

"language is an incredibly powerful encoding of thoughts and information but it's actually not a powerful encoding of what the 3d physical world that all animals and living things live in and if you look at human intelligence so much is beyond the realm of language language is a lossy way to capture the world"

Fei-Fei Li elaborates on the limitations of language as a sole means of understanding intelligence, arguing that it is insufficient for encoding the complexities of the 3D physical world. She posits that language is a "lossy" representation, implying that much of reality is not captured by words alone.

"one way to think about it is we do a lot of language processing and we use that to communicate and the high level ideas etc but when it comes to navigating the actual world we really really rely on the world itself and our ability to reconstruct that"

Martin Casado uses an analogy to differentiate between language processing and spatial understanding, suggesting that while language is useful for high-level communication, navigating the real world relies heavily on direct perception and reconstruction of the environment. This emphasizes the practical necessity of spatial intelligence.

"but fundamentally physics happens in 3d and interaction happens in 3d navigating behind the back of the table needs to happen in 3d composing the world whether physically digitally needs to happen in 3d so fundamentally the problem is a 3d problem"

Fei-Fei Li asserts that the fundamental nature of physical interactions and the world itself is 3D. She argues that any AI system aiming to truly understand and interact with reality must operate within this 3D framework, as physics and spatial composition are inherently three-dimensional.

"and so for many things that are spatial you need to provide that information to the computer so that you can actually navigate in 3d space and so 2d video is great if it's a human because we already can turn it into 3d but like for any computer program it'll need to be 3d"

Martin Casado explains why 3D representation is crucial for AI systems, particularly robots, stating that 2D video alone lacks the necessary depth information for accurate spatial navigation. He contrasts this with human perception, where the brain can infer 3D from 2D, highlighting the distinct requirements for computational systems.

"but that's what makes the digital virtual world incredible with this technology which we should talk about it's the combination of generation and reconstruction suddenly we can actually create infinite universes some are for robots some are for creativity some are for socialization some are for travel some are for storytelling it suddenly will enable us to live in a multiverse way the imagination is boundless"

Fei-Fei Li envisions a future where the combination of generative AI and reconstruction capabilities allows for the creation of limitless digital universes. She suggests this technological advancement will enable a "multiverse way" of living, expanding human imagination and application across various domains.