Issue #91 Architecture Edwin Dong, Lalit Julapalli
Linear Attention For Transformers (Kimi Linear), Simply Explained
Free editionPRO edition
We’re excited to share that we recently updated our website to a new, modern, and more visually pleasing look. It sports both dark and light modes; you can check it out at aibutsimple.com.
Quick Note: The login on this new page is completely separate from the beehiiv login. It is ONLY for courses. Everything is still hosted on the old website, which can be found by clicking any post on the new one.
We have added an exciting new addition: courses! 3 course options are now available, all of which are coming soon.
We’re also in the process of launching an upskilling/training program designed for businesses seeking to expand their AI expertise. If your organization is interested in learning more, please reach out to us at edwin@aibutsimple.com.
This week’s PRO issue includes a Colab notebook that dives into the heart of linear attention, investigating KDA (Key-Aware Delta Attention) and it’s inner workings. We benchmark the speeds of quadratic and linear attention.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Today’s LLMs are built on the core principles of the attention mechanism, the key component of how LLMs generate and analyze text.
Attention has evolved over time as researchers and developers have found methods of optimizing its performance, which can be seen through the quality upgrades seen in transformer outputs.
This article will not be a formal literature review but rather a gateway into understanding the results and rationales behind evaluating new attention mechanisms and the inner workings of the Kimi Linear Attention mechanism, published in a 2025 paper by the Kimi team.
It will also be quite helpful to first find a familiarity with how transformer attention works.
We have recently published a two-part series of articles breaking down how modern transformers work, along with current advances that have improved the effectiveness of using attention.
This edition is for PRO members.
