Issue #91 Architecture Edwin Dong, Lalit Julapalli
Linear Attention For Transformers (Kimi Linear), Simply Explained
We’re excited to share that we recently updated our website to a new, modern, and more visually pleasing look. It sports both dark and light modes; you can check it out at aibutsimple.com.
Quick Note: The login on this new page is completely separate from the beehiiv login. It is ONLY for courses. Everything is still hosted on the old website, which can be found by clicking any post on the new one.
We have added an exciting new addition: courses! 3 course options are now available, all of which are coming soon.
We’re also in the process of launching an upskilling/training program designed for businesses seeking to expand their AI expertise. If your organization is interested in learning more, please reach out to us at edwin@aibutsimple.com.
Today’s LLMs are built on the core principles of the attention mechanism, the key component of how LLMs generate and analyze text.
Attention has evolved over time as researchers and developers have found methods of optimizing its performance, which can be seen through the quality upgrades seen in transformer outputs.
This article will not be a formal literature review but rather a gateway into understanding the results and rationales behind evaluating new attention mechanisms and the inner workings of the Kimi Linear Attention mechanism, published in a 2025 paper by the Kimi team.
It will also be quite helpful to first find a familiarity with how transformer attention works.
We have recently published a two-part series of articles breaking down how modern transformers work, along with current advances that have improved the effectiveness of using attention.
A Review of Basic Transformer Attention
Breaking down attention to its very simplest component, we must understand that attention is calculated for each word/token by providing information about only words/tokens that came before it. This is what’s called the process of causal masking, illustrated below:
