# How DeepSeek’s Architecture Actually Works #117 Aug 31 Architecture https://www.aibutsimple.com/p/how-deepseek-s-architecture-actually-works In 2026, it's common to hear stories of autonomous coding agents with Claude Fable 5 or GPT-5.6 burning through hundreds of dollars in API credits in under an hour. Frontier capabilities have been scaling impressive heights, but the compute costs of this rise have made running these models at scale economically brutal. Those costs represent the fork in the road for most users between their wallet and their needs. It is also precisely why DeepSeek’s architecture captured the industry's attention. By reapproaching traditional LLMs on the compute weight and storage aspects, DeepSeek became the example for how smarter architectural choices can deliver frontier performance at a fraction of the cost. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/25853f52-02ff-4b5f-8187-5b82389c1549/DeepSeek_Architecture__8_.png) Today, we’ll provide a visual breakdown of the three architectural pillars that make the cost-cutting possible: 1. DeepSeek Mixture-of-Experts (MoE): Why fine-grained expert segmentation and shared expert routing yield higher specialization with lower active parameter counts. 2. Compressed Attention Mechanisms: How Multi-Head Latent Attention (MLA) and its evolution into dual-path hybrid attention beat standard quadratic-scaling mechanisms across 1M token context windows. 3. Group Relative Policy Optimization (GRPO): Adjusting the RLHF dramatically cuts training memory overhead. THE WEEK, RANKED | [Qwen 3.8-Flash-Next by Alibaba](https://huggingface.co/Qwen/Qwen3.8-Flash-Next?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | ▲ 4.1KHF | | [VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning](https://huggingface.co/papers/2608.26105?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | ▲ 253HF | | [GLM 5.3 Flash by Z.ai](https://huggingface.co/zai-org/GLM-5.3-Flash?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | ▲ 1.5KHF | | [Community Claude plugins (Anthropic)](https://github.com/anthropics/claude-plugins-community?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | ▲ 2.1KGH | | [WeMM-Embedding by Tencent](https://github.com/Tencent/WeMM-Embedding?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | ▲ 901GH | [ALL ENTRIES](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works)RANKED BY [BASIS, OUR TRACKER](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) | Presented by | ![viso.ai](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7046a56a-d96b-4eb6-b960-b56acac3079d/image.png) | ### Build a Vision AI agent with only one prompt ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/dad3e74e-8361-424d-aca6-99c6d574757f/Viso_Now_-_Tube-map-hub-spot-header_02_1200x600.png) If you've ever built a computer vision model, you know the real cost was never the model. It was everything before it: collecting images, labeling them by hand, and waiting weeks before you found out if any of it worked. Viso Now skips that step. Describe a scene or a problem in plain language, and the system builds a computer vision agent directly from that description. No labeled dataset, no separate model trained for each new use case. It runs on an engine called VGI-1, part of a thesis [viso.ai](http://viso.ai/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) calls Visual General Intelligence: the idea that a system can reason about a scene it has never been trained on, instead of needing a model for every variation. We just launched, and we'd rather you tried it than took our word for it. [Try it Now](https://viso.ai/?utm_campaign=486336754-CREATOR-0826-AIBUTSIMPLE&utm_source=aibutsimple&utm_medium=Twitter) #### Fine-Grained Segmentation through MoE The number of parameters is the first metric developers see from the model card. Here are total parameter counts for DeepSeek’s newer models: - DeepSeek V4 Pro, 1.6T - DeepSeek V4 Flash, 284B That’s quite a few parameters, so DeepSeek relies on MoE to dynamically adjust which subset of networks and parameters to use for each query. Despite both boasting massive amounts of parameters, only about 3% (pro) and 5% (flash) of them are used for a single forward pass. This leverages the MoE design pattern: the belief is that information is sparsely/non-uniformly represented. Information can be found within very specific subsets of nodes and parameters. In comparison to a simple MoE, DeepSeek splits feed-forward networks into many smaller experts and activates a lot of them at once (e.g., 16 out of 160). ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ab7170b-ff63-4002-8a11-e6d3837b99c2/DeepSeek_Architecture__1_.png) This allows it a higher flexibility and specialization, since more precise combinations of its capabilities can be chosen while maintaining its active compute (FLOPs) and memory budgets. DeepSeek also isolates dedicated shared experts to always remain active, as they can be used for universal patterns like grammar and formatting. Routed experts benefit from this choice because they don’t have to waste capacity on rounding out the edges. There’s been a problem where the same subset of networks receives most of the attention, which means most of the experts sit idle, causing processing bottlenecks. Some techniques to combat this: - Auxiliary Load-Balancing Loss Penalty spreads out the workload across unused experts. It degrades performance since the router tries to pick the best expert every time. - Auxiliary-Loss-Free Load Balancing, which applies a dynamic bias per expert depending on real-time traffic. This is what DeepSeek uses. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a68dddf2-be8b-4a88-b1a8-ab6c80ddff55/DeepSeek_Architecture__2_.png) Underloaded experts get a boost when routing is processed to keep GPU utilization balanced. --- #### Multi-Head Latent Attention (MLA) Multi-Head Latent Attention (MLA) is an attention mechanism designed to drastically reduce memory usage during inference by compressing the Key-Value (KV) cache. Traditionally, LLMs used Multi-Head Attention (MHA), which splits input sequences into parallel attention heads so that the model can simultaneously compute attention for different positions. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2606bee0-77ba-49ec-b2a1-4d3e86cb00ac/DeepSeek_Architecture__4_.png) The problem is that computing this attention for different positions requires caching distinct key and value vectors (KV vectors) for every head at each step, which creates a memory bottleneck when context windows grow to 1M tokens long. MLA solves this by a smart procedure of compressing and decompressing these KV vectors. Instead of caching full KV vectors, a single joint low-rank compression of the KV vectors is stored. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2a158d03-b47a-4453-a557-388a825ee6b5/DeepSeek_Architecture__5_.png) When this vector representation is needed during the attention step, a learned projection matrix “uncompresses” or projects the vector back to the proper key/value space to restore accurate multi-head attention calculations. From using Rotary Position Embeddings (RoPE), which encodes relative distances and appends positional information, MLA splits each vector into two parallel branches: 1. A compressed content vector 2. A small, uncompressed positional vector. This allows the model to apply positional encodings directly to the positional path without disrupting the low-rank compression matrix. This preserves the content vector’s information when it needs to be decompressed. For a more in-depth explanation of RoPE, read more in [our article about Modern Transformers](https://newsletter.aibutsimple.com/p/a-deep-dive-into-modern-transformers-rope-activation-functions-quantization). --- #### Group Relative Policy Optimization (GRPO) To train these massive models, Reinforcement Learning with Human Feedback (RLHF) is used alongside algorithms like Proximal Policy Optimization (PPO). Standard PPO requires training two separate models and keeping two other frozen ones. One of them is the Policy Model, which is the one being fine-tuned and generating answers, and the other is the Critic Model, which evaluates the future expected reward for every individual token. A Reward Model is used to determine how “good” the response is with respect to the prompt (the reward). There’s also a Reference Model, which is a frozen copy of the Policy before RL tuning. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f9fc5c6f-3a9a-44e3-a099-1b78462f9242/DeepSeek_Architecture__6_.png) The benefit of the Critic Model is that of variance reduction. Specifically, the variance of the update steps. Since the Critic calculates the expected reward at all tokens, we can compare it to the actual reward to compute the advantage. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/549dc00d-c654-4328-86ad-03920809e827/image.png) The advantage (A) determines which tokens should appear with increased probability (At > 0). Here, Rt is the accumulated reward for every token from t onwards, which stays fairly constant (one scalar emitted at the end), and V(st) is the expected accumulated reward. The state st is made up of the prompt and every token so far. For instance, if a prompt is easy for the Policy Model to answer well, the Critic Model will learn to expect a high reward. When the high reward arrives, the advantage is smaller since the V is closer to R, reducing the variance of the advantage estimates used to update the Policy. The problem with having an additional Critic Model is that when holding it in-memory (plus their optimizer states), it consumes as much training memory as the Policy Model itself. Limiting the available memory means limiting the context length that can be trained upon in RL. GRPO removes the Critic Model and instead replaces it with a group of responses from the Policy Model that are weighed against the relative average. The variance reduction angle is satisfied, as good responses won’t significantly outpace the group’s average reward. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/852bb898-0be6-43b9-a247-29d20e460a19/image.png) Above, Ai is the response’s advantage, ri is the response’s reward, and mean(r) is the group’s average reward. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4459e9d9-063c-4486-933e-2dd27a02244e/DeepSeek_Architecture__7_.png) If a prompt is difficult, then all of the responses should be attributed to a lower score, leading to a low average score. However, since each response’s individual score is compared to the average, the good-scoring responses have a positive advantage, meaning that the probabilities of the tokens in the good responses are scaled to appear more often. Essentially: - PPO: “We need a baseline to estimate much better/worse each token's outcome was than expected.” - A critic learns that baseline with another model. - GRPO estimates it from multiple responses to the same prompt instead of using a separate model. A key difference is that the advantage in PPO is calculated for each token’s likelihood, while the advantage in GRPO is calculated for each response’s tokens. --- #### The Bottom Line DeepSeek’s cost revolution came from experimenting on each major compute/memory sink. - Swapping large feed-forward layers for fine-grained MoE routing and shared experts, reducing the total computational expense on each query. - Shrinking KV caches and the associated GPU memory down to low-rank latent projections. - Betting that statistical baselines can effectively replace Critic Models while keeping their benefits When a model like DeepSeek-V4 Flash can deliver frontier-grade performance at just $0.14 per 1M input tokens and $0.28 per 1M output tokens, it proves that clever mathematical framing beats brute-force GPU scaling every single time. That's it for this week's issue of AI, but simple. See you next week! --- Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan Want to reach 10000+ AI researchers? Advertising Inquiries: [WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=how-deepseek-s-architecture-actually-works) Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).