Mixture of Experts (MoE) Models

Imagine that you needed to build a house. You would not just hand the entire project to one person and expect them to do everything from plumbing to roofing. Instead, you would hire a team of specialists, each with the right tools and expertise for a specific job. When a new task comes in, you would decide which expert is the best suited to handle the task.

This is what Mixture of Experts (MoE) models do—they are ML models based on the premise of power in numbers. The notion behind them is similar to ensemble models: to assemble a group of candidates to produce the best output from all of the candidates.

Essentially, MoE models are a way to make neural networks more efficient and scalable without sacrificing performance—sometimes even improving it. Instead of having one large, dense neural network trying to process all inputs, an MoE model uses multiple smaller, specialized neural networks called "experts."

Then, a gating network (or router) dynamically determines which expert, or combination of experts, is most suitable for processing a given input. By implementing both components, MoE models have scalable parameter counts while keeping the computational cost for each input relatively low.

MoE allows models to be pretrained with far less compute, meaning you can scale up a huge amount with the same compute budget as a dense model. Additionally, this also means that an MoE model should achieve the same performance as its dense counterpart much quicker during pretraining.

In terms of real-world applications, MoEs are used everywhere, from the top LLMs (DeepSeek r1, switch transformers) to computer vision and reinforcement learning.