# Linear Attention For Transformers (Kimi Linear), Simply Explained

#91 Mar 2 Architecture

https://www.aibutsimple.com/p/linear-attention-for-transformers-kimi-linear-simply-explained

We’re excited to share that we recently updated our website to a new, modern, and more visually pleasing look. It sports both dark and light modes; you can check it out at [aibutsimple.com](https://www.aibutsimple.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained). Quick Note: The login on this new page is completely separate from the beehiiv login. It is ONLY for courses. Everything is still hosted on the old website, which can be found by clicking any post on the new one.

We have added an exciting new addition: courses! 3 course options are now available, all of which are coming soon.

We’re also in the process of launching an upskilling/training program designed for businesses seeking to expand their AI expertise. If your organization is interested in learning more, please reach out to us at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

Today’s LLMs are built on the core principles of the attention mechanism, the key component of how LLMs generate and analyze text.

Attention has evolved over time as researchers and developers have found methods of optimizing its performance, which can be seen through the quality upgrades seen in transformer outputs.

This article will not be a formal literature review but rather a gateway into understanding the results and rationales behind evaluating new attention mechanisms and the inner workings of the [Kimi Linear Attention mechanism](https://arxiv.org/pdf/2510.26692?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained), published in a 2025 paper by the [Kimi team](https://www.kimi.com/en?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ff79e2d9-a23b-4ee0-9cd2-8e9f33dfa23b/3_Visuals_for_Kimi_Linear__3_.png)

It will also be quite helpful to first find a familiarity with how transformer attention works.

We have recently published a two-part series of articles breaking down how modern transformers work, along with current advances that have improved the effectiveness of using attention.

- [Modern Transformers Article 1: Basics, Flash Attention, GQA, Optimizations](https://aibutsimple.beehiiv.com/p/a-deep-dive-into-modern-transformers-optimizing-attention?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)
- [Modern Transformers Article 2: RoPE, Activation Functions, Quantization](https://aibutsimple.beehiiv.com/p/a-deep-dive-into-modern-transformers-rope-activation-functions-quantization?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)
- Also recommended: [Key-Value Caching](https://aibutsimple.beehiiv.com/p/llm-kv-caching-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

### A Review of Basic Transformer Attention

Breaking down attention to its very simplest component, we must understand that attention is calculated for each word/token by providing information about only words/tokens that came before it. This is what’s called the process of causal masking, illustrated below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/27c06db6-030f-4ef3-942c-687a7990f52d/Causal_Masking_and_Attention_Scores.png)

By crunching together our keys, values, and queries matrices, along with their weight matrices, we calculate an attention score for each word, which then allows models to generate future words by projecting onto a vocabulary dataset of vector embeddings.

Looking at the image above, which contains the formula of attention, you should pay attention to the attention matrix. That is the technical term for the grid that is shaded with different hues of blue, representing the magnitude of the attention scores.

These attention scores denote “how important” a certain word is in relation to another.

For this reason, we can expect the strong, deep blue color along the diagonal, corresponding to high attention scores. This is expected behavior since words will correspond to themselves very highly.

It’s important to take note of this, as the actual paper discusses these attention matrices in detail. Here is another one to see a tighter example:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2341edda-9d72-4b0d-b8ec-ef11f366d934/3_Visuals_for_Kimi_Linear.png)

Another important detail to note down is that this grid expands like a square. In particular, for an attention matrix that represents n words/tokens, we need a matrix with n2 grid-entries, or equivalently, an n2 number of calculations.

This represents the primary bottleneck of our standard attention: compute resources required will scale quadratically.

Not just compute resources needed to perform the actual matrix multiplications in the attention formula, but also the memory resource needed to retain information about the respective matrices.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

---

### The Goal of the Paper

When discussing the concept of a Linear Attention Model, the objective is to develop a mechanism that scales linearly with respect to the number of tokens (n) that are being given to the model.

As you can expect, however, these systems may be more efficient, but they underperform when subject to accuracy evaluations.

The Kimi Linear Attention paper outlines a novel approach to discovering linear attention by taking advantage of a hybrid design that combines the efficiency of linear attention with the accuracy of full attention.

They found that this produces a model that, for the first time, outperforms the normal full attention model under fair evaluations.

### Helping the Model Forget

Think of calculating attention as calculating a series of states. These states will contain the attention information of how words relate to each other. Implementing this in linear attention gives the following formula:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4d7f9d38-213b-4d3b-8254-2dff37b8d626/image.png)

Here, k and v are the key and value vectors. As you can see, the linear states serve to keep “appending” or adding on to previous states. Due to this, these states only increasingly grow in magnitude, essentially “locking in” features of the past.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f2f5c05f-6865-43ab-a84f-808f285e49f6/3_Visuals_for_Kimi_Linear__1_.png)

The problem here is that information that could have been absorbed long ago may interfere with future, more relevant information.

The other underlying issue is that this matrix is of a fixed size, which makes it less interpretable as more associations are expected to be established over time.

Think about this like taking notes on a blank index card. At first, the note-taking experience is great, but as you keep going to new lectures and trying to write in it, you don’t gain any more space, and you may find yourself having to write over parts of it. This is exactly what is happening with our state management.

So how do we fix this? The answer is surprisingly intuitive. Instead of just adding new information to the state, we can first scale down the existing state before lumping on whatever’s new. This scaling factor is called a decay gate, and it looks like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1fa452d7-9ab7-4f69-8eb1-4deca334d47c/image.png)

Above, alpha (α) is the decay gate, a number between 0 and 1. Over time, older associations begin to fade, prioritizing what’s new to the table. Mathematically, this process is governed by the decay gate being multiplied over and over.

However, this isn’t the perfect solution, primarily because of the fact that this scaling factor, alpha, generalizes all information within a state to be of equal importance.

Since words/tokens can be relevant over the course of many paragraphs and sections, we don’t want our model to display forgetting qualities when remaining on the same string of topics and thoughts.

For this reason, applying this blanket decay gate works best as a rough approximation. That’s the very core of utilizing a Gated DeltaNet (GDN).

What if we could apply the same idea, but account for each of the dimensions of a matrix (rows and columns of information)?

The researchers of the Kimi Linear Attention model developed a new type of GDN, where an appropriate scaling factor was applied for each dimension.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ba63b8f-1fce-4bff-8f74-92aed64eb848/image.png)

Here, diag(α) is a diagonal matrix of all the different scaling factors. The diagonal matrix term is the new dimension-wise gate.

Rather than one number scaling the entire state, it's a diagonal matrix where each entry along the diagonal corresponds to a different “forgetting” rate for a different feature dimension. These are updated along with the states, so they’re also learning from patterns in the data.

The (I - βtktktT ​) term is what’s known as the Delta Rule. Before writing new information in, the model first erases the old association for the current key.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bba11b02-05be-405a-96c4-ad58182a7f99/3_Visuals_for_Kimi_Linear__4_.png)

This term essentially asks, "What does my memory currently think about this key, and how wrong is it?" It then subtracts out that old association.

Then the βtktktT ​ is added back in at the end. Analogously, the Delta Rule is like a smart eraser, helping you decide what to keep on the index card.

The Decay gate and Delta Rule combined make Kimi Delta Attention (KDA), the main contribution of the paper. These two features help the model recover most of its natural updating abilities from full attention.

### Hybrid Architecture

With KDA, the architecture faces one more hurdle: long-term memory management.

The advantage of full attention is that when each new token is calculated, every single token from the very beginning is also recalculated, which is lost with linear attention.

Of course, the decay gate and delta rule don’t enforce perfect precision, but the paper discusses a smart workaround.

Kimi Linear combined KDA layers with periodic full attention layers at a fixed 3:1 ratio, with three KDA layers for every one full attention layer. The KDA layers handle the bulk of computation efficiently, while the full attention layers act as “checkpoints.”

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/78fd2130-e879-465c-9beb-ac544b7bfeb8/3_Visuals_for_Kimi_Linear__2_.png)

Therefore, this combination preserves the model's ability to look back precisely when it needs to.

---

### Learn to engineer production-ready AI agents with memory systems, sub-agent pipelines, and reusable skills in this free webinar—a live 2-hour deep dive.

Go beyond autocomplete. In this [hands-on session](https://www.vpdae.com/redirect/9bzaruehd0i8tmp4lxn6eqaxhpc?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained), you’ll design reliable AI agents using structured memory, task-isolated sub-agents, and composable skills.

See a real bug-fix pipeline in action and learn patterns you can apply with Claude, LangChain, CrewAI, or AutoGen.

[Reserve Your Spot!](https://www.vpdae.com/redirect/9bzaruehd0i8tmp4lxn6eqaxhpc?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

---

### Notable Results

They found exceptional results: the Kimi team ran all three models, full MLA attention, GDN-H (Pre-KDA), and KDA, on the exact same training setup with a 1.4 trillion token input size. The only difference was the attention mechanism itself.

The KDA mechanism, in tandem with their ratio of layers, proved effective over nearly all of their evaluations, which included a mix of text answers, math olympiad problems from the AIME and USAMO tests, and RLHF.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a8253732-99ac-4385-9db0-3b147ef76f70/image.png)

On general knowledge benchmarks, Kimi Linear came out on top. It scored 51.0 on MMLU-Pro against MLA's 47.2 and GDN-H's 47.9. On long-context tasks at 128k tokens, it hit 84.3 on RULER versus MLA's 81.3.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d4aed605-0774-4f84-baf2-2e76de5126ac/image.png)

The efficiency gains were just as prominent. At 1 million token context lengths, the Kimi Linear model is 6.3 times faster than full attention per output token. It also uses 75% less KV cache memory, making it memory-lean and computationally viable.

What they found most interesting was in the RL testing.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6d8584cc-843a-4986-8c3b-ce7c092421af/image.png)

The Kimi Linear Model outperforms the Full Attention Model through the entire training process. This is quite revealing because it suggests that the model is actually effective over time and not just producing results at the last snapshot.

The reason that it’s important is because having results compound like this gives us more transparency about the attention model and can help us identify dips and the causes of those dips in performance.

### The Bottom Line

Kimi Linear Attention is a meaningful step forward for the field, showing that accuracy and efficiency don’t have to be trade-offs of each other. Through the innovative approach of mixing the attention layers, both are achieved in parallel.

There’s still research to be done on the RL tasks. The theoretical groundings of why Kimi Linear performs better on RL tasks are not fully clear yet, and the work remains open source to encourage collaboration.

In our [PRO article](https://aibutsimple.beehiiv.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained) for this week, we’ll investigate the implementation of this attention model and run real-time tests against full attention, so be sure to check that out!

As LLMs are increasingly deployed in agentic settings and expected to handle long conversations, tool use, and extended reasoning, the KV cache bottleneck only gets tighter. In this space, Kimi Linear has provided intriguing results, inviting further exploration.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-attention-for-transformers-kimi-linear-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team