Issue #85 Inference Edwin Dong, Lalit Julapalli
A Deep Dive into Modern Transformers: Optimizing Attention
Free editionPRO edition
This week’s PRO issue includes a Colab notebook with implementations of different attention algorithms from scratch—while comparing and visualizing them.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Artificial intelligence has been around since the 70s, but a team of researchers at Google published a paper in 2017 that transformed the landscape: Attention Is All You Need.
With the adoption of the transformer architecture came the concept of self-attention, a strategy of predicting tokens that are fed into the transformer.
The main idea of these transformers is very widely studied. Feed in a chain of tokens, process them using attention, then obtain a probability distribution over the next possible tokens to sample from.
However, modern transformers used in LLMs like GPT-5, Gemini 3 Pro, and DeepSeek R1 have undergone significant updates and optimizations. Specifically, the attention mechanism has been heavily updated.
In this issue, we’ll dive deep into attention and its modern variants, showing exactly how they improve performance.
This edition is for PRO members.
