# The Architecture of Kimi K3.1 Pro, Simply Explained

#126 Nov 16 Architecture

https://www.aibutsimple.com/p/the-architecture-of-kimi-k3-1-pro-simply-explained

Welcome back to AI, but simple, your AI research beacon with visual explanations.

We're going to do a deeper explainer article this week on a very impressive model release that topped the open-weight LLM benchmarks: Kimi K3.1 Pro.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/619f8384-c888-4431-b081-8e50f8b1f776/Kimi_K3.1_Pro__Simply_Explained__5_.png)

When it launched in October, [Artificial Analysis](https://artificialanalysis.ai/articles/kimi-k3-1-pro-launch-benchmarks?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) scored it 55 on its Intelligence Index, fourth overall.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e3b3a433-473d-4daf-b308-b7909cd3734b/image__393___1_.png)

Artificial Analysis Intelligence Index, Launch Day

Kimi recently released its weights on [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3.1-Pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained), which makes it the highest-scoring open-weight model on the index, with a 3 point gap between the best proprietary model.

[Kimi Model Card](https://huggingface.co/moonshotai/Kimi-K3.1-Pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained)

Moonshot describes the model as Kimi K3 with much more reinforcement learning with an interesting architectural change.

The 24 full-attention layers now use DeepSeek Sparse Attention (DSA).

That change fits a pattern across open models.

Qwen3-Next mixes linear attention with a few gated full-attention layers, and [Zhipu's GLM-5](https://arxiv.org/abs/2602.15763?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) adopted DSA earlier this year.

Let’s uncover what’s going on behind the scenes of such a frontier model.

The week, ranked

| [Benchmarking the Capability and Reliability of System One Models (SO-Arena)](https://huggingface.co/papers/2611.15777?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) | ▲137HF |
| [Yogurt, an open-source, fully local AI notetaker app](https://github.com/FrigadeHQ/yogurt?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) | ▲1.6KGH |
| [Learning Heterogeneous Tactile Representation for Dexterous Manipulation](https://huggingface.co/papers/2611.12806?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) | ▲104HF |
| [I built a swimmable underwater world in one 97 KB HTML file with Fable 5.2](https://www.reddit.com/r/ClaudeAI/comments/1xw376f/i_built_a_swimmable_underwater_world_in_one_97_kb/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) | ▲8.7KRD |
| [Introducing Gemini 4 Hydrogen, Google's Newest Model](https://deepmind.google/blog/introducing-gemini-4-hydrogen/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) | ▲1.3KRD |

[All entries](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained)Ranked by [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained)

| PRESENTED BY | ![LangrangeCloud](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/29dca96b-2b8b-4530-a07e-abb6714fc69a/image.png) |

#### Stop paying for steps your agent repeats

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/be1da377-5668-4fad-9485-d6a0e1a97db1/Lagrange_1200x600.png)

If you’ve built an agent with Claude Code, Hermes or Codex, the hard part was getting it to work.

Making it cheaper and faster is a whole other project: tuning prompts, testing changes and hoping nothing breaks.

[Lagrange Cloud](https://lgr.tensorblock.co?utm_source=ai-but-simple&utm_medium=newsletter&utm_campaign=main_2026-11-09) deploys the agent you already have and studies every run. After about five, it suggests a fix, and you replay the change before deciding if it ships.

Get free credits on us with code AIBUTSIMPLE.

[Deploy your agent →](https://lgr.tensorblock.co?utm_source=ai-but-simple&utm_medium=newsletter&utm_campaign=main_2026-11-09)

#### Kimi K3.1 Pro Architecture

First, it’s important to note that although the model is open weights, K3.1 Pro has no dedicated research paper describing their methodology.

So, we have to rebuild the architecture mostly from the custom code and config files.

The first clue is a name.

Inside the config, the text model is registered as “KimiLinearForCausalLM,” so K3.1 Pro is built on the code from Moonshot's [Kimi Linear](https://arxiv.org/abs/2510.26692?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) research.

It also lists the types of all 93 transformer blocks in the model.

- For every 4 transformer blocks, the first, second, and third blocks use Kimi Delta Attention (KDA), while the fourth is a Gated DeepSeek Sparse Attention (Gated DSA).
- Note that 93 isn’t divisible by 4, so the last block repeats the gated DSA.
- This adds up to 69 KDA layers and 24 gated DSA layers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59d008d5-7287-4936-9247-51f9063d98bb/Kimi_K3.1_Pro__Simply_Explained__6_.png)

The gated DSA is a change from their K3 model, which used full-attention gated multi-latent attention (MLA).

A few other details come from the same file.

Only the very first layer uses a regular feedforward block, while the other 92 use MoE with 896 experts, 16 active, plus 2 shared experts.

And what about quantization and weight storage?

Only the routed experts are quantized to 4-bit MXFP4, while attention, shared experts, router, and vision encoder are high precision.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3ab919d0-195e-437f-ac70-0dd770dceadf/MXFP4_Quantization__2_.png)

- The experts turn out to make up most of the model (roughly 2.7T out of 2.78T parameters).
- In MXFP4, weights are grouped into blocks of 32, stored as 4-bit floating-point numbers. Every block shares one 8-bit scale, a power of two.

Activations are computed in MXFP8, which is the same idea as MXFP4, just with 8 bit floats.

The attention layers are the most interesting architecture change compared to K3, so let’s take a look at them closely.

### Full Attention is the Bottleneck

KDA is a form of linear attention. We covered Kimi’s linear attention in one of our previous articles, which I’ll link here:

- [Kimi Linear attention, Simply Explained](https://www.aibutsimple.com/p/linear-attention-for-transformers-kimi-linear-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/181d1032-47b3-4e13-97f6-db2021359bae/3_Visuals_for_Kimi_Linear__5_.png)

As a refresher, each KDA head keeps one memory matrix St and updates it for every new token:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/45e128ca-9284-4895-bc75-44c7c8462256/image.png)

The attention state has a decay mechanism from erasing and fading. St fades a little using ɑt, where each channel fades at its own rate.

It then erases part of what it currently stores for the key kt and writes the new value vt in its place. βt is a dynamic writing strength.

Since St has a fixed size, a KDA layer costs the same at one or a million tokens.

However, a fixed-size memory can’t store everything, which is a reason to keep non-linear attention.

In this case, K3.1 Pro uses gated sparse attention for every 4th layer.

Let’s take a look at how a gated DSA works layer below.

### DeepSeek’s Popular Sparse Attention

DSA comes from [DeepSeek-V3.2](https://arxiv.org/abs/2512.02556?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained), released in December 2025.

The idea is simple: most earlier tokens probably don't matter for the current one, so find the ones that do with a cheap scan, then run full attention only on those.

The cheap scan is a small module called the lightning indexer. For the current token t, it gives every earlier token s a score:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/51e1ce91-cc14-424b-8f4b-67dad9854670/image.png)

Each indexer head j takes a dot product between a small query qt,j and a small key ks.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2448f0b-51b5-4659-b67f-4d943891fced/image.png)

Lightning Indexer in Green. Note that K3.1 Pro uses no RoPE.

The ReLU drops negative scores, and the weights wt,j decide the importance of each head.

The result, It,s, is a relevance score per earlier token, and the model keeps the top 2048 tokens to process for attention.

At a full million-token context, that is 512 times fewer cached entries read per token in each DSA layer.

K3.1 Pro's config gives the indexer 32 heads of size 128.

The weight files add four new indexer pieces to each DSA layer: a query projection, a key projection, a key norm, and the projection that produces the head weights wt,j

Two caveats with DSA:

1. The indexer still scans every earlier token, meaning its cost still grows with context length. It’s cheaper than full attention, with just few small heads.
2. DSA doesn't shrink the KV cache. Any token may be chosen later, so every token's compressed vector still has to be stored with an indexer key. DSA saves on reads and computes, not on storage.

Well, what about the “gated” part of gated DSA?

The other half of the name carries over from K3. Each attention layer ends with a sigmoid gate on its output:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6684acf2-8801-4f50-ad36-b36bdffa578e/image.png)

Here, õt is the attention output, and xt is the layer's input.

The sigmoid turns each channel into a number between 0 and 1, so the current token decides how much of each channel to let through.

Qwen's [Gated Attention](https://arxiv.org/abs/2505.06708?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained) paper found that this kind of gate makes training more stable and removes attention sinks, where heads dump most of their attention onto the first token.

The weight names show that each DSA layer keeps K3's attention projections, norms, output gate, and output projection and only adds the indexer.

So a Gated DSA layer is really K3's Gated MLA layer with a lightning indexer in front of it. Interesting, right?

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/76ecc15e-fa3a-42d9-8088-7339bddf6e8a/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__13_.png)

Illustrative. There should be a lightning indexer and a top-k selector that will pick from the latent space. The output after attention will have a sigmoid gate.

Now, let’s also talk about the activation function Kimi K3.1 uses.

#### Kimi’s SiTU-GLU Activation Function

The feedforward blocks, including every expert, use an activation function Moonshot introduced in Kimi K3 called SiTU-GLU (Sigmoid Tanh Unit GLU).

It’s the modern replacement for SwiGLU, which Kimi K2 and most current LLMs use.

So why not SwiGLU? SwiGLU splits inputs into 2 branches.

- The gate branch computes z = Wgx and passes it through Swish = z * σ(z)
- The up branch computes u = Wux and leaves it alone, then is multiplied with the gate branch.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ad4bc65-cc71-4ff9-aef3-d7370b045bbe/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a9405531-2f82-4226-858d-982dc49cc699/Kimi_K3.1_Pro__Simply_Explained__1_.png)

The issue here is that both factors can grow without limit.

If z and u happen to be large in the same channel, their product becomes a huge outlier. That’s a risk in low-precision arithmetic, where large numbers can overflow.

This was observed empirically at 2.8 trillion parameters, where the model’s matrix multiplication chain produced exploding activations.

SiTU-GLU keeps numerical values in a safe range. It uses a soft cap based on a hyperparameter β, which uses tanh: β*tanh(x / β).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/447606f3-56cc-4757-bd36-03778311ffec/Kimi_K3.1_Pro__Simply_Explained__3_.png)

For small inputs, this is almost exactly x, so nothing changes.

The key here is that we flatten out outliers. For large inputs, it can never go past β.

Kimi applies the cap to the linear part of the Swish and, separately, to the up branch, giving us 2 β values:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/08760852-d76e-4749-9baf-d0cc4153e051/image.png)

K3.1 Pro, like K3, uses β1 = 4 and β2 = 25, so no output can ever exceed 4 × 25 = 100.

That's it for this week's free version of AI, but simple. Get deeper analysis, ad-free reading, and full access to the full archive with PRO. See you next week!

---

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-architecture-of-kimi-k3-1-pro-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com)
