# Mixture of Experts (MoE) Models #54 Jun 9, 2025 Architecture https://www.aibutsimple.com/p/mixture-of-experts-moe-models Imagine that you needed to build a house. You would not just hand the entire project to one person and expect them to do everything from plumbing to roofing. Instead, you would hire a team of specialists, each with the right tools and expertise for a specific job. When a new task comes in, you would decide which expert is the best suited to handle the task. This is what Mixture of Experts (MoE) models do—they are ML models based on the premise of power in numbers. The notion behind them is similar to ensemble models: to assemble a group of candidates to produce the best output from all of the candidates. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5a9f699b-ecd9-49b8-a808-24ef3a8d1db8/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__720_x_600_px___2_.png) Essentially, MoE models are a way to make neural networks more efficient and scalable without sacrificing performance—sometimes even improving it. Instead of having one large, dense neural network trying to process all inputs, an MoE model uses multiple smaller, specialized neural networks called "experts." Then, a gating network (or router) dynamically determines which expert, or combination of experts, is most suitable for processing a given input. By implementing both components, MoE models have scalable parameter counts while keeping the computational cost for each input relatively low. MoE allows models to be pretrained with far less compute, meaning you can scale up a huge amount with the same compute budget as a dense model. Additionally, this also means that an MoE model should achieve the same performance as its dense counterpart much quicker during pretraining. In terms of real-world applications, MoEs are used everywhere, from the top LLMs ([DeepSeek r1](https://arxiv.org/abs/2501.12948?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mixture-of-experts-moe-models), [switch transformers](https://arxiv.org/abs/2101.03961?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mixture-of-experts-moe-models)) to computer vision and reinforcement learning. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/723ffa33-3deb-4635-9957-b2b0b18a45f4/image.png) --- Big thanks to our partners for keeping this newsletter free. If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you’ll enjoy. ### Bridge the gap with student loans June’s here—are your college costs covered? With final aid letters rolling in, now’s the time to see if there’s a gap. Federal aid might not cut it, but [private loans](https://secure.money.com/pr/gb7d27155e28?apxa3=1010&utm_medium=email&utm_campaign=XIAEVJHTGJ&utm_source=beehiiv&utm_content=versionD&_bhiiv=opp_d067dffc-b641-4d0e-b188-5d5a8af6f4d4_886cf9ce) can. Schools suggest applying early, so don’t wait. Check out Money’s top picks for low-rate, hassle-free student loans. [SEE MONEY’S TOP LENDERS](https://secure.money.com/pr/gb7d27155e28?apxa3=1010&utm_medium=email&utm_campaign=XIAEVJHTGJ&utm_source=beehiiv&utm_content=versionD&_bhiiv=opp_d067dffc-b641-4d0e-b188-5d5a8af6f4d4_886cf9ce) --- ### Architecture An MoE model primarily consists of two key elements: smaller components called experts and a routing mechanism known as the gating network. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d97d8c5e-f2d7-43ea-a36c-01185d00af98/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__2_.png) #### Experts Experts are typically smaller, independent neural networks such as feed-forward networks (FFNs) or other more complex networks. Each expert is trained to become proficient in handling specific types of patterns in data or smaller sub-tasks within the broader general problem. - In transformer MoE architectures that process tokens, each expert typically learns syntactic information on a word level rather than specific topics like “math” and “English.” In traditional transformers or other non-MoE models, the FFN layers used are dense since all of their parameters are activated in an input pass. In MoE models, sparse FFN layers or other sparse expert networks are used, only activating a portion of the total parameters spread across the whole network. - The underlying idea is to only activate one or just a few specific experts at a time that are most relevant for a task, reducing computational cost. #### Gating Network A gating network/router is a mechanism that selects which expert to use, and in the case of transformers, determines which tokens are sent to which expert. This component is also a small neural network—typically a linear FFN layer with a softmax activation function. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2181de5-52a5-493f-a7aa-78efc3487a22/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust.png) The gating network takes input data, transforms it using the linear layer, and then outputs a distribution of probabilities across all available experts using softmax. Just like sampling the next token in a transformer, the expert or experts that have the highest probabilities are chosen; they are deemed most suitable for the current input. For instance, in the image below, two tokens are passing through a switch transformer: the router sends the token “More” to the second expert (FFN 2) and sends the token "Parameters” to the first network (FFN 1). ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18f360f4-a9da-4467-af1e-d746d9387436/image.png) How to route inputs to an expert is one of the big decisions when working with MoEs—the gating mechanism itself is a huge determining factor of an MoE’s performance. The gating mechanism is made up of learned parameters and is pretrained at the same time as the rest of the network. ### Operational Flow The operational flow within an MoE layer is as follows: First, an input instance (such as a token embedding for NLP) is fed into the MoE layer. The gating network then processes the input and transforms it using an FFN or other network, passing the FFN’s output to a softmax activation—this generates a set of probabilities, indicating the relevance of each expert to the input. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/926045a0-cc4c-42ce-b9bd-043247ac4974/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__720_x_600_px_.png) Based on the scores of the gating network, a small subset of experts (usually top-k, where k is a small integer like 1 or 2) are chosen to handle the data. The other experts that are not chosen are inactive, conserving compute. The activated expert(s) then process the data, and a weighted sum using the scores from the gating network as the weights is produced. Multiple experts’ individual outputs are combined in the weighted sum. The output of the MoE layer is passed to the next portion of the network, and multiple MoE layers may be used to further improve performance—for transformers, we can replace every FFN block with an MoE block. ### Advantages and Disadvantages Overall, MoE models offer several notable benefits: - Only a fraction of total parameters are used, reducing computational load (FLOPs). Compare this to dense models (such as fully connected models), which use all parameters. - Models can be easily scaled to large parameter counts in the hundreds of billions or trillions—without a proportional increase in the cost per input. - In certain cases, the specialization of the individual experts can lead to better performance. However, despite their advantages, MoE models also present certain challenges: - MoEs are more complex/difficult to train as there are more components than FFN layers. - Load balancing is a challenge in training since ensuring that all experts receive a sufficient and balanced amount of training data is difficult—if the gating mechanism consistently favors a few experts, others may not learn as quickly. - The overall performance of MoE models is highly dependent on the gating network's ability to learn effective routing—performance takes a hit when routing is poor. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mixture-of-experts-moe-models): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mixture-of-experts-moe-models) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mixture-of-experts-moe-models) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team