# Looped Transformers, Simply Explained

#123 Oct 19 Architecture

https://www.aibutsimple.com/p/looped-transformers-simply-explained

Welcome back to AI, but simple, your AI research beacon with visual explanations.

We’re going to do a deeper explainer article this week on a topic that quickly became one of the most popular LLM architecture tricks: the [looped transformer](https://app.aibutsimple.com/topics/looped-transformer?family=technique&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained).

It’s a simple but clever trick that has seen adoption in LLM research and, more recently, industry, being a part of frontier models like GPT-6 Astra.

| The week, ranked |
| [Waypoint-OPD, transferring policy shifts with gated On-Policy Distillation](https://huggingface.co/papers/2610.21601?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) | ▲ 660 HF |
| [Give your AI agent real phone calls over MCP, with outcomes and transcripts returned automatically](https://call4.me/?utm_source=aibutsimple&utm_medium=newsletter&utm_campaign=secondary_oct2026) | Presented by[call4me](https://call4.me/?utm_source=aibutsimple&utm_medium=newsletter&utm_campaign=secondary_oct2026) |
| [A fully opne stack for generalist computer use (OpenGCUA)](https://huggingface.co/papers/2610.22200?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) | ▲ 315 HF |
| [Muse Spark Open, the most capable US open-weight model to date](https://www.reddit.com/r/LocalLLaMA/comments/1x6r9pz/introducing_muse_spark_open_the_most_capable_us/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) | ▲ 1.3K RDDT |
| [Open source scion (a faster git for large systems) alternative for AI agents](https://github.com/feder-cr/git-ai?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) | ▲ 1.1K GH |
| [All entries](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained)Ranked by [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) |

Before reading further, here are some nice things to keep in mind.

When we refer to a transformer block, we’re talking about a module with attention, feedforward layer, normalization, and residual connections.

A stack is a sequence of transformer blocks.

You should also know what a KV cache is. It’s essentially storing the attention keys and values for earlier tokens to prevent recomputation.

To get a better grasp of these topics (if you’re not aware), we would recommend taking a look at our past articles on transformers and KV caching.

- [KV caching, Simply explained](https://www.aibutsimple.com/p/llm-kv-caching-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained)
- [Transformers, Simply Explained](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained)

#### Looping Blocks Together

So first, what is a looped transformer?

A Looped Transformer is essentially an architectural tweak where intermediate representations are passed through the same transformer blocks multiple times (“loop”) instead of once.

The key idea here is that the weights stay the same across these passes.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cc824ddb-f4cb-481c-b93c-bfb16b16c9aa/Looped_Transformers.png)

This tweak isn’t anything new, and the idea was first explored in a [2018 paper](https://arxiv.org/abs/1807.03819?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) titled “Universal Transformers.”

Let’s start with an example.

In a standard transformer, every token passes through a stack of blocks exactly once, and each block in the stack has its own weights.

If you want a deeper (and more robust) model that does more computation per token, you add more blocks.

The problem is that each block brings a new set of weights with it. This can get expensive, fast.

So what if we kept just one block and applied it over and over? This is the seemingly simple idea behind a universal transformer.

We’ll be able to progressively refine the hidden states using the same blocks without adding weights, and the great part is how every step works just like a regular transformer layer.

We can write one step as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/aa890180-f190-4d48-bafe-6643862dc335/image.png)

Here, Ht holds the hidden states of all the tokens after step t, and Blockθ is the repeated block with shared weights.

The extra term Pt is a clever detail. Since the block is identical on every step, the model doesn’t know which step it’s on.

So before each step, the Universal Transformer adds an embedding that encodes both each token's position and the current step number, which tells the block where it is in the loop.

If you "unroll" the loop, a Universal Transformer that runs for six steps seems just like a six-layer transformer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5f46a6bc-43e6-48bd-b451-205f3ba505f1/image.png)

The universal transformer architecture

The difference is that all six layers share one set of weights, so the model obtains the depth of six blocks while only storing one.

#### RNNs and Recurrent Depth

If you worked with RNNs in the past, the looping or “recurrent depth” idea should be somewhat familiar.

The core idea of RNNs is also weight sharing, reusing the same weights from previous iterations to gradually modify sequence data.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0c6feedd-111f-4bc9-a92d-e07fded1de9c/RNN_sequence_forgetting.png)

The difference is the direction of the loop. An RNN loops over time, reading tokens, updating its hidden state, then reading the next token using the updated state. Earlier info gets carried forward one step at a time.

A looped transformer loops over depth instead. It takes each token's hidden state and keeps refining it by running it through the same block again.

Since it’s a transformer, whenever tokens can be processed in parallel, they are processed in parallel (e.g., prefill, training, etc.). The authors describe the model as a block of parallel RNNs running side by side.

One RNN is used per token, and they all share the same weights.

While like RNNs, looped transformers are more capable since each looped step has access to attention, so a larger piece of context is available at every step (RNNs have a small hidden state).

An additional benefit of looped transformers is that the number of steps is dynamic.

A regular transformer passes a representation through the same amount of layers, no matter the difficulty of the input.

A looped model can simply add more layers when a problem needs them.

| PRESENTED BY | ![ai, but simple pro](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59bb099e-a38a-445e-8f0d-b6cd300aa85d/_UPDATED__AI__but_simple_Pro_banner__1_.png) |

#### Industry Info and Insights

Our [AI intelligence layer](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) identified looped transformers as rising over a month ago, and here is a preview over the info it uncovered.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3b09f42c-d568-4b37-857a-d35f5ece8056/image.png)

##### Who’s using it

Microsoft stated on a publicly accessible web page that OpenAI has been using Looped Transformers in the GPT-6 series.

Cornell University and IBM Research pretrained LPT and h-lpt, looped Transformers whose first recursion writes the cache the later recursions share.

##### Positioning

- Looping and depth growth act as complementary mechanisms for inducing and scaling iterative computation in large language models.
- Its listed variants are Loop Memory Attention and Looped Language Models, and its parents are the Transformer and Ouro.

##### Background

- The idea of deepening a network by repeating one shared block instead of stacking new layers was introduced as the Universal Transformer by Dehghani and colleagues in 2018.

You can check out a free preview of the looped transformer topic page that we surfaced [here](https://app.aibutsimple.com/login?family=technique&next=%2Ftopics%2Flooped-transformer&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained), and you can unlock our whole database with AI, But Simple PRO.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/86d2e5d3-ca74-4e1e-aa58-945c43dd040f/068-ascii-muted-pro-billboard-560x290-7VA4Q50Z.png)

With [PRO](https://newsletter.aibutsimple.com/upgrade), unlock access to our AI research layer. It synthesizes emerging techniques and hot topics across thousands of papers into a single screen.

- Get benchmarks, relation graphs, authorship, industry adoption, and open questions for every technique.
- Perform smart searches across our vast database of techniques, papers, and code.

[Get AI Research Intelligence](https://www.aibutsimple.com/pro?utm_source=primary_ad)

#### Adaptive Halting

If all tokens loop the same number of times, a simple word ("the") gets exactly as much computation as the key word in a hard question.

But some tokens probably need more thinking than others. So, can the model decide for itself how long to loop?

The Universal Transformer tackles this with adaptive halting.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/91b2d39b-a774-48e7-9405-536c372cc8e6/Looped_Transformers__1_.png)

After each step, a small learned layer looks at each token's hidden state and outputs a halting probability for how “ready” that token is to stop.

The model keeps a running total of these probabilities for each token.

Once a token's total crosses a threshold, that token stops looping, and its hidden state is left alone, while the other tokens keep going until they also stop or reach a maximum number of steps.

#### Looping In Today’s Language Models

Although the Universal Transformer was built for tasks like translation and looped a single block, today’s looped language models loop a whole stack of blocks.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1691ff3d-38d1-4b41-9f3b-45c0dbfc46e1/image.png)

Modern Looped Transformers such as Ouro stack multiple layers in one timestep

Note that there are also drawbacks to looping.

Looping still requires the same amount of computation per token, going through the same amount of work as if it went through regular transformer blocks.

Looping doesn't save KV cache memory either. You might expect a block to reuse its saved keys and values on the second pass.

However, the hidden state entering the block on the second pass is different from the state seen the first time, with different KVs, so both sets have to be stored.

In fact, there’s a point where, with a long enough input, and depending on the specific looped architecture and how states are cached, the KV memory increase might surpass the weight saving.

So what looping really does is it trades parameter memory for compute and KV cache.

If you’re interested in reading the full universal transformer paper, you can find it below:

[Read The Universal Transformer Paper](https://arxiv.org/abs/1807.03819?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained)

I’ve also compiled below some models that explore the Universal Transformer's idea of choosing how long to loop:

- ByteDance's Ouro [[arxiv]](https://arxiv.org/abs/2510.25741?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) loops a whole stack up to four times and uses a learned exit gate after each pass to decide when to stop, much like adaptive halting.
- Mixture-of-Recursions [[arxiv]](https://arxiv.org/abs/2507.10524?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) (KAIST, Mila, and Google) uses a small router (the ones used in MoE) to give each token one, two, or three loops depending on its context.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/631613c3-e909-4bdc-9cc7-c4391fe62a93/image.png)

Mixture of Recursions applies a shared block at different positions (either once, twice, or thrice)

And on a side note, the KV cache issue is actively being tackled, with a [recent paper](https://arxiv.org/abs/2610.02383?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained) where only the first pass saves a full KV cache, and later passes read from that shared cache, keeping just a small cache of its own for the most recent few tokens.

### Wrapping Up

A looped transformer uses the simple trick of passing data through a transformer block or stack repeatedly to reduce stored parameter counts.

It uses weight sharing across depth, and is similar to RNNs, as outlined in the Universal Transformer paper.

It gives a smaller model the computation of a much deeper one, but it doesn't save compute or, by default, KV cache memory, which is why so much recent work focuses on the cache.

That's it for this week's free version of AI, but simple. Get deeper analysis, ad-free reading, and full access to the full archive with PRO. See you next week!

---

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=looped-transformers-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com)