# Nanbeige4.3-33B, Simply Explained

#125 Nov 2 Foundation

https://www.aibutsimple.com/p/nanbeige4-3-33b-simply-explained

Welcome back to AI, but simple, your AI research beacon with visual explanations.

Nanbeige4.3-33B was released this past week and is the new open-source flagship from Nanbeige LLM lab.

[Nanbeige4.3-33B on HuggingFace](https://huggingface.co/Nanbeige/Nanbeige4.3-33B?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained)

It is a representation of modern open-source LLM performance at the 30B weight class, and it shows a result that’s surprising in the midst of all the LLM architecture optimizations we’ve seen recently.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e0041043-226d-4d8b-88bd-c566573b1e49/Nanbeige4.3-33B__Simply_Explained__3_.png)

In this issue, we’ll unpack and visually explain Nanbeige’s choice to completely revert to a completely standard transformer architecture, despite its previous model being a [looped transformer](https://www.aibutsimple.com/p/looped-transformers-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained).

The week, ranked

| [Google's new frontier model: Gemini 4 Carbon](https://www.reddit.com/r/singularity/comments/1xhtwjm/gemini_4_carbon/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) | ▲1.6KRDDT |
| [Introducing GPT-6.1 Terra](https://www.reddit.com/r/singularity/comments/1xg4hs6/introducing_gpt61_terra/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) | ▲777RDDT |
| [An AI assistant for your android phone](https://github.com/wangzan101/Mika?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) | ▲2.7KGH |
| [PDRVLA: Vision-Language-Action Model with Physical Dynamics Representations](https://huggingface.co/papers/2610.37381?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) | ▲149HF |
| [Towards free-form Interactive World Generation](https://huggingface.co/papers/2610.36729?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) | ▲122HF |

[All entries](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained)Ranked by [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained)

#### The Big Picture

Nanbeige 4.3-33B was released without a technical report, so publicly available information only included benchmark results.

To take a peek at what’s inside this model, we reverse engineered the architecture by taking a look at the checkpoint itself and rebuilding it.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bd316175-b82d-498a-ae50-2529b8ea1637/Nanbeige4.3-33B__Simply_Explained__4_.png)

At the highest level, Nanbeige4.3-33B does what every decoder-only LLM does:

- It turns each token into a vector (through embeddings)
- Refines that vector through a stack of 52 transformer blocks
- And turns the result into an output probability for every possible next token.

The checkpoint confirms all of this, and if you wanted to check for yourself, the config file lists dimensions, while the index file contains layer information and reference to weight files.

Here is some additional info:

- There are 471 tensors: three for the input and output, plus the same nine in every block, adding up to 33.1 billion parameters.
- The nine tensors are exactly what’s used in a standard Llama implementation (there’s no custom code)

If you've seen a Llama block before, you already know the below info, but here are the details.

First, we normalize the token's embedding vector, then pass it through grouped-query attention (GQA) with rotary position embeddings (RoPE), and sum the output with a residual addition of the original vector before norm.

The first section’s output gets normalized a second time, then passes through a SwiGLU feedforward network, with another residual connection wired the same way.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/90794352-7a7f-4751-a622-0d560899911f/Nanbeige4.3-33B__Simply_Explained.png)

Both normalizations are RMSNorm, and the FFNs have no bias term.

After 52 of those blocks, a final norm and an output head turn the vector into probabilities over a vocabulary of about 167K tokens.

Just to note, the output head and the embedding table are two separate matrices.

So this is just a standard architecture, nothing interesting. Let’s investigate why the authors designed it this way.

### Wide or Shallow?

Let's start with the shape of the model.

A budget of 33 billion parameters can be spent on depth (more transformer blocks) or on width (a bigger vector inside each block).

A good comparison of the Nanbeige model is with Qwen3-32B, which has almost exactly the same number of parameters.

Qwen3-32B stacks 64 blocks around a vector of size 5,120, while Nanbeige stacks only 52 blocks with a vector size of 7,168.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0e2f9b4a-5bf9-4eb8-a28c-f079d577b2d1/Nanbeige4.3-33B__Simply_Explained__1_.png)

So why would the shape matter?

The answer stems from grouped-query attention. In both models, every block stores 8 key heads and 8 value heads with a fixed 128 size per token in the KV cache, however wide the vector is.

The larger the vector, the more parameters needed for the Q, K, and V weight matrices, which are not cached anyways.

So the cache grows with the number of blocks, not the width.

That actually gives Nanbeige a KV cache 19% smaller per token than Qwen3-32B for the same parameter count.

The part of attention's compute that grows with context length shrinks even more, by about 29%, because Nanbeige also has fewer query heads per block (56 vs 64).

Fewer blocks also means fewer steps that have to run one after another.

Every generated token passes through all the blocks in order, and when the model is split across several GPUs, each block ends with two rounds of communication between them.

With 52 blocks, that is 104 rounds per token instead of 128.

### What Nanbeige Leaves Out

The most revealing thing about Nanbeige’s vanilla architecture is the parts that are “missing.”

Most recent LLMs come with modern architecture tweaks that are heavily adopted across the industry. Here are just a few:

- QK-norm: Qwen3 and many recent models normalize queries and keys before comparing them, which stops attention scores from growing out of control during training. Nanbeige? None.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/92a54017-03ef-4a48-9219-ef46acab25f1/Nanbeige4.3-33B__Simply_Explained__2_.png)

- Mixture of experts: MoE models such as Qwen3-30B-A3B split the feedforward network into experts and route tokens to a few of them, so only a fraction of the weights run per token. Nanbeige is actually dense, and all 33B parameters run for each token.
- Hybrid attention: Qwen3-Next swaps most of its attention layers for linear attention, allowing for fixed memory for long context, and keeps full attention in only 1/4 of its layers. Its cache is 25 KB per token, against Nanbeige's 213 KB.
- Latent attention: DeepSeek's MLA used in its models compresses keys and values into a small latent vector before caching them, another way to shrink the cache.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b418a624-cb70-4395-8cce-7457bfd312b3/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__12_.png)

But how can we tell that it’s missing these elements? We simply look at the config lines and custom code files of the model.

For instance, Nanbeige's looped 4.2-3B model seems like a plan 22-layer Llama model, but it contains “num_loops=2” in the config and custom code file.

We can also confirm that Nanbeige4.3-33B has no loops from the files that were provided on HuggingFace.

When we compared the query weights of all 52 blocks, no two were similar (max cosine similarity of 0.07), so there isn’t a loop saved as copies either.

| PRESENTED BY | ![ai, but simple pro](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59bb099e-a38a-445e-8f0d-b6cd300aa85d/_UPDATED__AI__but_simple_Pro_banner__1_.png) |

Our [AI intelligence layer](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained) identified Nanbeige’s release the hour it came out. Unlock our whole research database with AI, But Simple PRO.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/86d2e5d3-ca74-4e1e-aa58-945c43dd040f/068-ascii-muted-pro-billboard-560x290-7VA4Q50Z.png)

With [PRO](https://newsletter.aibutsimple.com/upgrade), unlock access to our AI research layer. It synthesizes emerging techniques and hot topics across thousands of papers into a single screen.

- Get benchmarks, relation graphs, authorship, industry adoption, and open questions for every technique.
- Perform smart searches across our vast database of techniques, papers, and code.

[Get AI Research Intelligence](https://www.aibutsimple.com/pro?utm_source=primary_ad)

#### Why Stay Standard?

So why would a lab that just released a looped transformer build its flagship this way?

One reason is to analyze what looping actually saves. Nanbeige4.2-3B has 22 transformer blocks, looped once, getting 44 blocks of computation from 22 blocks of weights.

That saves memory for parameters, but each pass still costs compute and keeps its own KV cache.

For a 3B model on a small device, weight memory is often the bottleneck.

For a 33B model at long context, the cache cost dominates, and looping doesn’t provide the performance to justify a bigger cache.

The Nanbeige team also noted this, stating that looping offers a better quality-compute trade-off at smaller model sizes.

The other reason is what the model is actually made to do, which is to be a teacher model.

It was actually built to generate training data and token probabilities for a new 3B release and small models after it.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c9392de3-ff2a-4fa6-84e0-709d94620c40/Token-level_teacher_feedback__4_.png)

Such a teacher has two requirements:

- It has to run cheaply at scale. A Llama-style checkpoint runs in vLLM and SGLang with no custom code, while the looped 4.2-3B needs its own modeling file plus vLLM-specific patches.
- Its output space has to be the same as the student's, providing a way to compare distributions (e.g., KL divergence). This means sharing the same tokenizer (byte-identical) across models.

Seen this way, the choices fit together.

The wide, shallow shape and GQA keep the cache and per-token latency low for the long rollouts a teacher generates, and the model also doesn’t need additional workarounds or custom files for its architecture.

That's it for this week's free version of AI, but simple. Get deeper analysis, ad-free reading, and full access to the full archive with PRO. See you next week!

---

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=nanbeige4-3-33b-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com)