Mixture-of-Experts Architecture Drives AI Intelligence and Cost Efficiency

NVIDIA AI Podcast · December 29, 2025 · Listen to Original Episode →

Original Title:

TL;DR

Mixture-of-Experts (MoE) architectures enable significantly larger and more intelligent AI models by activating only a fraction of total parameters, drastically reducing computational cost per token.
MoE models achieve higher intelligence scores with fewer active parameters, demonstrated by GPT-OSS (61 intelligence score, 5B active parameters) outperforming dense models (28 intelligence score, 405B active parameters).
The primary hidden cost of MoE models is inter-expert communication, necessitating high-bandwidth, low-latency interconnects like NVIDIA's NVLink to prevent GPU idle time and maintain efficiency.
Extreme co-design between NVIDIA's hardware (e.g., NVLink, GB200 NVL72) and AI model builders is crucial for managing MoE communication overhead and unlocking performance gains.
Advances in NVIDIA's infrastructure, such as scaling NVLink connectivity from 8 to 72 GPUs, have yielded 15x performance improvements for MoE models like DeepSeek-R1, reducing cost per token by 10x.
MoE's sparsity principle is applicable beyond language models to vision, video, and scientific AI applications, offering a path to smarter, more cost-effective AI across diverse domains.

Deep Dive

Mixture-of-Experts (MoE) architecture is revolutionizing AI by enabling significantly larger and more intelligent models without a proportional increase in computational cost. This approach activates only a fraction of the model's total parameters for any given task, drastically reducing inference costs and making advanced AI more accessible. The continued development of specialized hardware and interconnect technologies, like NVIDIA's NVLink, is crucial for overcoming the inherent communication complexities of MoE and further driving down the cost of intelligence generation.

The core innovation of MoE lies in its efficiency. Unlike traditional dense neural networks where all parameters are activated for every query, MoE models are composed of numerous specialized "experts." A routing mechanism directs incoming data to the most relevant experts, meaning a model with, for example, 405 billion total parameters might only engage 5 billion for a specific task. This leads to substantial cost savings in computation and energy. For instance, the intelligence score for a dense 405B parameter model might cost $200 to benchmark, while a comparable MoE model with only 5B active parameters achieves a higher score for roughly $75. This cost reduction per token is not just an incremental improvement; it enables the scaling of AI capabilities to previously prohibitive levels.

The implications of this cost efficiency extend beyond lower operational expenses. It fuels a virtuous cycle where hardware advancements enable more complex and capable AI models, which in turn demand and justify further hardware innovation. NVIDIA's extreme co-design approach, focusing on synchronizing hardware (like GPUs and NVLink) with AI model architectures and software stacks, is central to this progress. Technologies like NVLink are specifically engineered to facilitate high-speed, low-latency communication between the numerous experts within an MoE model. This is critical because the "hidden cost" of MoE is communication overhead; inefficient communication can lead to GPUs sitting idle, negating the cost savings. By providing a no-compromise network that allows every GPU to communicate with every other at full speed, NVIDIA's infrastructure enables MoE models to scale to hundreds of billions or even trillions of parameters while maintaining cost-effectiveness, as demonstrated by a 15x performance improvement on MoE models with a modest increase in total cost, resulting in a 10x reduction in cost per token.

Looking ahead, MoE is not just a trend for language models but a foundational concept likely to permeate other AI domains, including vision, video, and scientific computing. The principle of activating only necessary components for efficiency is universally applicable. As AI models evolve towards more complex reasoning, multimodal capabilities, and specialized scientific applications, the need for intelligent, sparsely optimized architectures will only grow. NVIDIA's commitment to co-design, from the microscopic signaling on wires to the macroscopic architecture of data centers and the software that orchestrates it all, positions them to support these future advancements, continuing to drive down the cost of intelligence while expanding its capabilities.

Action Items

Audit MoE architecture: Analyze 3-5 core models for communication bottlenecks and GPU idle time, identifying opportunities for NVLink optimization.
Implement MoE training strategy: Define parameters for expert routing and activation across 2-3 model layers to reduce computational cost.
Measure MoE inference cost: Track token generation cost for 5-10 representative queries, aiming for a 10x reduction compared to dense models.
Evaluate co-design impact: Assess the performance gains from NVLink and specialized hardware on 3-5 MoE models, quantifying cost-per-token improvements.
Draft MoE deployment guide: Outline best practices for managing communication overhead and GPU utilization for 2-4 common MoE model types.

Key Quotes

"we've all heard of neural networks and that's what these neural networks are they're neurons they're parameters they're components of an ai model and you know when ai got started and really became in the zeitgeist of the world the neural network was simply uh each parameter represented a neuron of the model and we heard about uh one billion parameter model and a 10 billion a 100 billion now trillion parameter models those are basically the neurons of the ai brain that you you activate when you when you ask chatgpt a question but something happened along the way as these models got smarter and smarter and smarter they naturally got bigger and bigger and bigger"

Ian Buck explains that traditional neural networks, the building blocks of AI models, grow in size with increasing intelligence. This growth means that for every query, all parameters (neurons) must be activated, leading to slower performance as models become more complex.

"so to make the ai cheaper or the tokens which is the piece of data that's flying through that eventually becomes a word on the screen the tokens cheaper let's only activate the neurons we need to activate and that's what mixture of experts is instead of having one big model we actually split the model up into smaller experts same number of total parameters but now we only ask the uh we train the model to only ask the experts that probably know that information along the way"

Ian Buck introduces Mixture of Experts (MoE) as a solution to the cost and speed issues of large neural networks. He clarifies that MoE models, while having the same total number of parameters, are divided into smaller "experts," and the model is trained to activate only the relevant experts for a given query, thereby reducing computational cost.

"so today most models today are achieving higher and higher intelligence scores by taking advantage of having more than lots of experts and able to ask have the model as it comes up with the answer ask only the right experts in order to get that to get the right answers to give you put some numbers behind it you know we have that llama 405b 405 billion parameter that's one big model you know on on leaderboards like artificial intelligence you mentioned you know it gets an intelligence score of about 28 28 is just a weighted score of the benchmarks they they tested sure but all 405 billion parameters are getting activated right now fast forward to like a modern uh open model like uh open ai's uh gpt oss model it has 120 billion parameters actually a little bit smaller in total parameters but when you ask it a question it only activates on the order of about 5 billion parameters"

Ian Buck provides a concrete example to illustrate the efficiency of MoE. He contrasts a dense 405 billion parameter model (Llama) that activates all parameters with a 120 billion parameter MoE model (GPT-OSS) that only activates approximately 5 billion parameters for a given query, highlighting a significant reduction in computation.

"the beauty of ai is that the algorithms that these researchers and scientists and companies like anthropic and open ai and everybody else have figured out is that they can just give it the data and they encourage the model to to sort of camp to to identify and create these little pockets of knowledge it's not prescriptive it's just the data that they're seeing and naturally clumps the activity of these different questions to different experts"

Ian Buck explains that the specialization of "experts" within an MoE model is not hard-coded but emerges organically through the training process. The AI, by processing vast amounts of data, learns to group related knowledge and tasks, assigning them to different experts without explicit human instruction.

"the idea of experts is not new in machine learning you know before ai there was an idea of creating you know combining multiple machine learning models together and how to do that with statistically to improve the accuracy uh there's all sorts of history and math around that applying it to ai though is is relatively new you know the early versions of of we now we now know where chatgpt they were mixture of experts models but they were not public publicly known"

Ian Buck clarifies that while the concept of combining multiple models, or "experts," for improved accuracy has existed in machine learning for some time, its application to large AI models like those powering ChatGPT is a more recent development. He notes that early versions of these advanced models utilized MoE architecture but were not widely publicized.

"the hidden tax moe and it's all about how those experts need and want to and need to communicate with each other in order to get moes to run efficiently those experts are all doing their math very very very fast and they all need to communicate with each other very very very quickly and one of the challenges with moes is and as we go and get sparser and sparser and sparser which makes the models more and more valuable and we're we're saving saving more and more cost is can we make sure that all that math is happening and all those experts can talk to each other without ever running going idle without ever waiting for that a getting waiting for a message"

Ian Buck identifies communication overhead as a significant "hidden tax" in MoE models. He explains that for MoE to be efficient, the numerous experts must communicate rapidly and seamlessly. The challenge lies in ensuring that these experts can exchange information without delays or idle periods, which would negate the cost savings.

"and that's what nvlink is in fact that chip that we built is specifically designed to make sure that every gpu and its all of its terabytes of second of of bandwidth can talk to every other chip at full speed and never compromise on the the maximum amount of io bandwidth we can get out of every gpu we did that with hopper with eight way and one of the big innovations and obviously it took a lot of engineering to make that 72 racks everyone of the 72 because i could everyone of those net that gpus at full speed no constraints and you can see that taking off you can see the benefit you know that allows people to go even further and build even bigger models"

Ian Buck highlights NVIDIA's NVLink technology as a critical solution for the communication challenges in MoE models. He explains that NVLink is engineered to enable every GPU to communicate with every other GPU at maximum speed without bandwidth limitations, which is essential for scaling MoE architectures and building larger, more capable models.

"this is the extreme co design that we do at nvidia and some of our the folks that i get to work with and probably watching this get to enjoy and we we work really really hard to continuously work on performance not just to have the fastest and be the fastest but also to reduce the cost because perform you talked about tokenomics if our just our software alone could increase performance by 2x you've now reduced the cost per token by 2x directly to the to the user and the customer or whoever was going to deploy this ai"

Ian Buck describes NVIDIA's "extreme co-design" approach, where they collaborate closely with AI model builders. He emphasizes that this collaboration focuses on optimizing performance not just for speed but also for cost reduction, noting that software improvements alone

Resources

External Resources

Books

"The Idea of Experts" by Ian Buck - Mentioned as a concept that has been around in machine learning before AI, concerning combining multiple machine learning models.

Articles & Papers

"Deepseek Coder LLM" (Deepseek) - Discussed as the first world-class MoE-based model that competed and demonstrated leading intelligence scores.
"GPT-4" (OpenAI) - Mentioned as an example of a model that uses MoE architecture.
"Llama" (Meta AI) - Mentioned as an example of a model that uses MoE architecture.
"Llama 2" (Meta AI) - Mentioned as an example of a model that uses MoE architecture.
"Mixture-of-Experts (MoE) Frontier Models" (NVIDIA Blog) - Linked as a resource to learn more about MoE models.

People

Ian Buck - Vice President of Hyperscale and High Performance Computing at NVIDIA, guest on the podcast discussing Mixture-of-Experts (MoE) architecture.
Jensen Huang - Presenter of NVIDIA's keynotes at GTC.

Organizations & Institutions

Anthropic - Mentioned as a company that builds AI models.
Meta AI - Mentioned as a company that builds AI models.
NVIDIA - Company that develops AI hardware and software, host of the NVIDIA AI Podcast.
OpenAI - Mentioned as a company that builds AI models.

Websites & Online Resources

GTC (GPU Technology Conference) - Recommended as a developer conference to learn more about AI, GPUs, and NVIDIA's technology, with presentations available online.
Leaderboard (Artificial Intelligence) - Used as a reference point for comparing intelligence scores of AI models.

Other Resources

Mixture-of-Experts (MoE) - Architecture enabling smarter AI models with reduced compute and cost by activating only necessary parameters.
Tokenomics - Concept referring to the cost of generating tokens in AI systems.
NVLink - NVIDIA technology enabling high-speed communication between GPUs, critical for efficient MoE model performance.
Extreme Co-design - NVIDIA's approach of collaborating with AI model makers to optimize hardware, software, and models for maximum utility and performance.