Spatial Intelligence: AI's Next Frontier Beyond Language Models

Key Quotes

"deep learning is in some sense the history of scaling up compute when i graduated from grad school i really thought that the rest of my entire career would be towards solving that single problem which is a lot of ai as a field as a discipline is inspired by human intelligence we thought we were the first people doing it it turned out that was also simultaneously doing it so marble like basically one way of looking at it it's the system it's a generative model of 3d worlds right so you can input things like text or image or multiple images and it will generate for you a 3d world that kind of matches those inputs"

Fei-Fei Li explains that deep learning's progress has largely been driven by increased computational power. She introduces Marble as a generative model for 3D worlds, capable of creating these worlds from various inputs like text or images. This highlights the current capabilities and foundational concept of their new model.

"i think one is just there is a lot more data and compute generally available i think the whole history of deep learning is in some sense the history of scaling up compute and if we think about alexnet required this jump from cpus to gpus but even from alexnet to today we're getting about a thousand times more performance per card and then we had an alexnet days and now it's common to train to train models not just on one gpu but on hundreds or thousands or tens of thousands or even more so the amount of compute that we can marshal today on on a single model is is you know about a million fold more than we could have even at the start of my phd"

Fei-Fei Li elaborates on the role of compute in deep learning's advancement. She contrasts the computational resources available during the AlexNet era with today's capabilities, noting a million-fold increase. This emphasizes the significant growth in compute power as a key enabler for current AI models.

"i think the role of of academia and especially in ai has shifted quite a lot in the last decade and it's not a bad thing it's a sense of it's it's because the technology has has grown and emerged right like five or ten years ago you really could train state of the art models in the lab even with just a couple of gpus but you know because that technology was so successful and scaled up so much then you you can't train state of the art models with a couple of gpus anymore and it's not a bad thing it's a good thing it means the technology actually worked but that means the expectations around what we should be doing as academics shifts a little bit and it shouldn't be about trying to train the biggest model and scaling up the biggest thing it should be about trying wacky ideas and new ideas and crazy ideas most of which won't work"

Justin Johnson discusses the evolving role of academia in AI. He explains that the success and scaling of AI technology mean academics can no longer focus on training the largest models with limited resources. Instead, Johnson suggests academics should prioritize exploring novel, unconventional, and potentially unsuccessful ideas.

"i do think spatial intelligence is complementary to language intelligence so i i personally would not say it's spatial versus traditional because i don't know what traditional means what does it mean i do think spatial is complementary to linguistic and uh and uh how do we define spatial intelligence it's the capability that allows you to reason understand move and interact in space and i use this example of the deduction of dna structure right and of course i'm simplifying this story but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in a 3d space to to eventually conjecture a double helix"

Fei-Fei Li defines spatial intelligence as a capability that complements linguistic intelligence. She clarifies that spatial intelligence involves reasoning, understanding, moving, and interacting within space. Li uses the example of deducing the DNA structure, highlighting how spatial reasoning about molecules and bonds was crucial to this scientific breakthrough.

"a transformer is actually not a model of a sequence of tokens -- transformers is actually a model of a set of tokens right the only thing that gives that that injects the order into it in in the transformer in the standard transformer architecture the only thing that differentiates the order of the things is the positional embedding that you get the tokens right so if you if you choose to give it a sort of one d positional embedding that's the only like mechanism that the model has to know that it's a one d sequence but all the models that are happening inside a transformer block are either token wise right so that either you have an ffn you have qkv projections like you have norm per token normalization all of those happen independently per token and then you have interactions between tokens through the attention mechanism but that's also sort of -- it's permutation variant so if i permute my tokens then the attention operator gets a permuted output in exactly the same way so it's actually a natively an architecture of set of tokens"

Justin Johnson clarifies the fundamental nature of transformers. He explains that transformers are inherently models of sets, not sequences. Johnson states that the ordering of elements in a transformer is primarily introduced through positional embeddings, and the internal operations are token-wise and permutation-invariant, meaning the architecture itself is designed for sets.