Issue #123 Architecture Edwin Dong
Looped Transformers, Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook where we’ll run our own experiments with Looped Transformers.
Please do read the full article before attempting the code tutorial.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Welcome back to AI, but simple, your AI research beacon with visual explanations.
We’re going to do a deeper explainer article this week on a topic that quickly became one of the most popular LLM architecture tricks: the looped transformer.
It’s a simple but clever trick that has seen adoption in LLM research and, more recently, industry, being a part of frontier models like GPT-6 Astra.
Before reading further, here are some nice things to keep in mind.
When we refer to a transformer block, we’re talking about a module with attention, feedforward layer, normalization, and residual connections.
A stack is a sequence of transformer blocks.
You should also know what a KV cache is. It’s essentially storing the attention keys and values for earlier tokens to prevent recomputation.
To get a better grasp of these topics (if you’re not aware), we would recommend taking a look at our past articles on transformers and KV caching.
This edition is for PRO members.
