# The Dragon Hatchling (BDH), Simply Explained

#105 Jun 8 Architecture

https://www.aibutsimple.com/p/the-dragon-hatchling-bdh-simply-explained

For nearly a decade, the Transformer architecture has been the undisputed frontrunner of modern deep learning.

However, as models scale into the trillions of parameters, the quadratic scaling O(n2 ) of the self-attention mechanism is a harsh limitation.

As a model’s context window grows, the Key-Value (KV) cache memory requirements explode, making an ideal "infinite context," where a model continuously learns like our brains do, practically impossible on standard hardware.

Additionally, recent [complexity theory proofs](https://arxiv.org/abs/2509.04664?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained) suggest that autoregressive attention is mathematically bound to hallucinate at predictable rates.

In response, recent research has seen a pivot toward "Post-Transformer" architectures.

- These models prioritize linear-time inference, constant memory bounds, dynamic state evolution, and the ability to learn continuously without forgetting.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/54cc7a4f-c30c-4b08-b529-8af17f7929af/The_Dragon_Hatchling__BDH___4_.png)

In the post-transformer field, we’ve seen the emergence of [State-Space Models](https://newsletter.aibutsimple.com/p/mamba-and-state-space-models-simply-explained), and linear transformer backbones such as the [Gated DeltaNet](https://arxiv.org/abs/2412.06464?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained), among other architectures.

One specific post-transformer architecture sticks out from the rest. Interestingly, The [Dragon Hatchling (BDH)](https://arxiv.org/abs/2509.26507?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained) (Kosowski et al., 2025) models its architecture after the human brain.

Created by a team of researchers at Pathway (including ex-Google Brain, Inria, and Mila), the BDH is promising for long-horizon reasoning and is a strong candidate for continual learning.

---

### What You’ll Learn

1. The biological inspiration behind BDH
2. The 2 fundamental operations enabling BDH reasoning
3. The 4-stage BDH inference process
4. BDH-GPU, a GPU-friendly BDH equivalent

### What’s Helpful to Know

- Continual Learning
  - A domain in deep learning research that focuses on models that have the ability to learn new information continuously without degrading or "forgetting" previous knowledge.
- ALiBi & [RoPE](https://newsletter.aibutsimple.com/p/a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)
  - Popular positional embedding techniques used to model relationships dependent on token positions. RoPE (Rotary Position Embedding) rotates vectors to encode relative distances, while ALiBi applies a penalty to naturally weaken distant tokens.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6ce79431-a9ff-4b1b-859e-a4813d10023e/RoPE__Rotary_Positional_Embedding__Positional_Encoding__3_.png)

- Truncated Backpropagation Through Time (TBPTT)
  - A training method used for recurrent models with a running memory state. TBPTT splits long sequences into smaller chunks to calculate weight updates, reducing GPU overhead at any given time.

---

### The Dragon Hatchling (BDH)

The Dragon Hatchling (BDH) takes inspiration from the biology of the human brain.

The brain is a large-scale graph-based system with 8*1010 neurons. Only certain groups of neurons are involved at once, as the brain operates sparsely in a distributed graph representation.

The paper describes BDH as an edge-reweighting distributed graph where computations occur locally at neuron-like nodes, while communication occurs along weighted edges (or synapses) that connect them.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/059e3cad-6c52-4cf3-adba-00f7b576eaef/image.png)

Reasoning emerges from local neuron and synapse updates according to biologically inspired interaction rules.

The model goes through stages of computation called kernels (which define how nodes compute and exchange information over time) that are cycled through by a scheduler.

This scheduler will cycle through 4 different kernels (essentially a 4-stage process) repeatedly during inference.

---

### Motivating the BDH Architecture

Before going over what these stages of processing look like, let’s first motivate the architecture.

First, we’ll need to understand what an implication is. An implication is a learned relationship of the form:

If A is true or active, then B is likely to be true or active.

It is the proposition asserting “If A, then B” (A → B).

BDH is a reasoning system built on two core operations that surround implications:

1. Modus Ponens (following implications)
2. Hebbian Learning (updating implications)

The first operation is Modus Ponens, and it’s the most basic rule in logic that follows implications. If A implies B, and A is true, B is also true.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b62353c0-fd07-4d15-8fb2-fbd5f75ffc18/The_Dragon_Hatchling__BDH___7_.png)

So Modus Ponens extends the implication of A to B given an observation (A is true) to deduct that B is also true.

For instance:

- It is raining, the ground is wet.
- It is raining.
- Therefore, the ground is wet.

The paper explicitly builds BDH around this idea. A synapse weight (connection) σ(i,j) is interpreted as the strength of an implication i → j, and it lies between 0 and 1.

- Think of something like: Rain → Wet Ground, Strength = 0.95

The paper writes:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9b398997-306c-40ad-a468-01fb559768ee/image.png)

Here, X(i) is the belief strength or activation at node i, A(j) is the incoming evidence at node j, and σ(i,j) is the strength of the implication between i and j.

This rule essentially says that if a belief in i exists (X(i) is high), and the implication i→j exists (the strength σ(i,j) is non-zero), we should increase the evidence in j (increase A(j)).

This is approximately Modus Ponens with a probabilistic form.

The second operation is Hebbian Learning, which has been referred to in the literature as “Neurons that fire together wire together.”

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b11f2cef-2d1c-4cda-aca7-2d59b9e370b9/The_Dragon_Hatchling__BDH___6_.png)

Hebbian Learning is applied with this rule:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0163bbb1-657c-4781-8268-fc0e4b5ed05d/image.png)

This roughly means if i and j repeatedly activate together, we should strengthen the implication connecting them.

Above, X(j) is still the belief strength at node j, but Y(i) is something like the output of a node i after it has survived selection and thresholding.

Overall, we can say Modus Ponens performs reasoning, while Hebbian learning updates the reasoning rules.

---

If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained) so we can keep doing what we do.

### [Get Ahead of AI With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/963eba00-ab31-4912-8c1f-1470a52f9ecc/aibutsimple_academy__2_.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Stay Ahead in AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained)

---

### The 4 Kernels of BDH

Before getting into the kernels, we should understand our state variables and graphs. The variables X(i), A(i), Y(i), and σ(i,j) form the model’s dynamic state.

The paper also defines 5 sparse graphs, which are the actual trainable structures of the model, shown below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/45ec2a7c-48cb-41a7-b8d9-f33263991ab7/image.png)

These hold the parameters that are used in training and inference and will establish the connectivity between the nodes.

#### 1. Inference from State

This is the Modus Ponens step that uses X, A, and σ. Currently active nodes spread evidence through existing implications.

We can represent this step as a summation over incoming neurons, accumulating evidence at a given node j with the following equation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/db51411c-bd56-4c5a-9fe4-c324801620b3/image.png)

This is a sparse matrix-vector multiplication.

The paper also includes this decay equation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1b578bc5-d988-47aa-8624-7006abb742ec/image.png)

This weakens connections over time unless reinforced. Without decay, everything eventually will connect to everything.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cdb0f027-bc31-430e-b5a6-fe1b7f4031b5/The_Dragon_Hatchling__BDH___8_.png)

#### 2. Reweighting of Synapse State

This kernel is the paper’s biological representation of attention.

Using Hebbian Learning, the model strengthens the connection between a source node and a destination node if their activations correlate. The edges will re-weight themselves over time based on activity patterns of neighboring nodes.

Hebbian Learning is what enables continual learning in the BDH.

- In contrast to transformers, which have their weights frozen during inference, a BDH has a dynamic synaptic state, which changes over time to incorporate new information.

We can interpret the re-weighting update as a change in the synapse state:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1e6a088a-d302-4acd-bfcc-67ab8f1d7df8/image.png)

Here, note that Gs is one of the trainable sparse graphs that determine which neurons are allowed to modify the connections.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/789ba053-3d59-40ee-92c4-2d6dc579b4b6/The_Dragon_Hatchling__BDH___9_.png)

Since Gs sparsifies the update, only a subset of edges participates, making the BDH what we call “scale-free”.

A scale-free network has many nodes with 1-10 edges, fewer nodes with 100 edges, and very few nodes with 10,000 edges, with no characteristic scale.

Given this property, the model won’t learn all possible connections between all neurons in a dense fashion, but instead, it will only learn implications along the sparse connectivity encoded by Gs.

#### 3. Neuron Replicator Dynamics

This stage is the network’s selection mechanism, implementing a non-linear threshold that determines which nodes will produce the next active state.

It computes Ye (excitatory) and Yi (inhibitory) through excitatory and inhibitory graphs.

Excitatory propagation uses Gye and follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c39e493a-f755-4f77-bd72-8c8b25d74583/image.png)

Inhibitory propagation uses Gyi and follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f3f77792-5abc-415c-9f55-ae7ba652593b/image.png)

The node computes Y as the difference between Ye and Yi :

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9b17624a-0eda-4cf9-9238-d18db5f205d6/image.png)

Here, + represents a ReLU activation, taking the max between the difference and 0. So here, many candidate nodes may have accumulated evidence, but only the strongest concepts survive.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a0becf05-1df5-430d-b63f-7d5fd40cac74/The_Dragon_Hatchling__BDH___10_.png)

#### 4. Inference from Parameters

The surviving nodes produce the next active state, allowing the next reasoning cycle to begin.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1a0b6bb0-0c43-462a-ab0d-fd55be998152/The_Dragon_Hatchling__BDH___11_.png)

The excitatory propagation is now:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f46a8f66-929e-4ece-9161-2d833f0b8e1a/image.png)

The inhibitory one is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd595e6f-9c95-44f2-98b5-f82b587677df/image.png)

And we combine them in the same fashion as with Y:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cfeb1f65-0e0e-47d5-9d7d-33ad260a9e5a/image.png)

X now becomes the activation vector for the next chain of reasoning, cycling through the 4 kernels we have just covered.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/705063ae-247e-43d7-9927-7162444bcc73/The_Dragon_Hatchling__BDH_.png)

The paper uses L layers (typically 8) for the model (and each layer undergoes the 4-step process), so a token effectively undergoes many kernel cycles before producing an output.

Specifically, a new token is ingested every 4L rounds (one full pass through L layers), and the result is retrieved 4L rounds later.

Note that all layers share the same set of weights for each of the parameter matrices.

---

### BDH-GPU, A GPU-Friendly BDH Architecture

Simulating huge amounts of nodes on a GPU is incredibly slow. We need to stick them in tensors.

BDH-GPU is the tensor-based mathematical equivalent of BDH, and it’s made to be the actual GPU-efficient implementation usable in production.

Instead of instantiating massive sparse graphs in memory (Gs, Gx, Gy,) BDH-GPU factorizes these connections into low-rank parameter matrices (E, Dx, Dy).

- Here, E represents an encoder matrix, and there are 2 decoder matrices Dx and Dy.

BDH-GPU is a state-space system defined by these low-rank matrices (E, Dx, Dy) that evolve through recurrent updates over time steps (tokens) and layers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d044884a-29f2-4606-87fe-f65d2db939f2/The_Dragon_Hatchling__BDH___3_.png)

The paper introduces a ReLU-lowrank block (consisting of E followed by a LayerNorm then Dx and a ReLU) that acts as a noise threshold to selectively reinforce certain signals or activation vectors.

BDH-GPU also features linear attention for scalability given high-dimensional inputs, maintaining an attention-state matrix that compresses past interaction information.

---

### A Detailed View Into BDH-GPU

We’ll now dive into the BDH pipeline. The model will process one token per time step.

First, an input token is passed through a standard embedding matrix to create an embedding vector. It enters the first layer of the model as an initial value state.

Then, for each layer l from 1 to L, we perform a sequence of computations surrounding three main state-space equations.

First, the activation (query) vector xt,l is generated by passing the previous layer’s output (yt, l-1) through matrix E, a layer norm, Dx and a ReLU activation. This is the ReLU-lowrank.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7732b2f2-6666-4b91-a035-054ff76442c2/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/48ebf400-1366-4d9f-9614-b8eb21927a51/The_Dragon_Hatchling__BDH___12_.png)

Here, xt,l acts as both the key and the query for the model’s linear attention. The value is an intermediate vector v* = LN(Eyt,l-1), which is simply xt, l before passing through Dx and ReLU.

Now that we have xt,l, the model needs to retrieve relevant historical context. BDH-GPU reads from a compressed, fixed-size attention state matrix ρt-1,l.

The query (xt,l) is multiplied by this matrix, then passed through a LayerNorm and the output decoder matrix Dy, followed by another ReLU.

Finally, it is multiplied element-wise with our original query xt,l to produce the final sparse output vector for this layer, yt,l.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0494f34c-106b-41e5-abbb-9d49554166c6/image.png)

Here, a* represents the output from linear attention, which equates to ρt-1,lxt,l.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ba39550-2d65-4b0e-8f7c-5152d34fb48d/The_Dragon_Hatchling__BDH___13_.png)

Before moving to the next token, the model must update its attention state ρ with the new information it just processed.

It does this by calculating the product of our key xt,l transpose and value v*, then adding it directly to the previous state matrix.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b55206bb-7594-49ad-8016-1b2edf44ad27/image.png)

Here, U is a diagonal or block-diagonal matrix for local rotation and damping (similar to ALiBi + RoPE), and is used to decay older connections in ρ naturally over time.

After a token passes through the architecture for all L layers, the final vector is projected by a linear layer to output the logits for the predicted token.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4ac13b59-132a-449c-8f0c-438a14043133/The_Dragon_Hatchling__BDH___14_.png)

For training, BDH-GPU uses Truncated Backpropagation Through Time (TBPTT), carrying its state matrix across minibatches. It is also designed for token-parallel training on GPUs.

Despite being a drastically different biological architecture, BDH-GPU matches GPT-2 in performance and scaling behavior, evaluated on language modeling and translation from 10M to 1B parameters.

BDH-GPU often demonstrates improved loss reduction per token with a simpler matrix structure, meaning it actually learns faster per data token than a standard Transformer.

---

### Conclusion

The BDH presents itself as a potential rival to the well-established transformer architecture with a few key optimizations modeled after the brain.

- It offers a sparse graph topology instead of dense connectivity.
- During inference, the synapse (edge) state σ(i,j) changes through Hebbian updates.
- The graph goes through recurrent reasoning cycles (X→A→Y→X) rather than performing a single feed-forward pass.
- Local computation, as each node only sees neighbors.

The interesting part of BDH is not merely that it uses neurons and synapses, but that reasoning, memory, and learning all occur within the same dynamical graph, rather than only learning during training as in standard Transformers.

---

### Post-Reading Resource

If you want to learn more about the BDH architecture and the larger debate around scaling and post-transformer architectures, I found this amazing [video resource](https://www.youtube.com/watch?v=hCjoMLuCuLQ&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained) discussing these topics.

It features AI researchers and experts, including Łukasz Kaiser (co-created ChatGPT and transformers), Llion Jones (cofounded Sakana AI), and many more.

One key quote from the video:

We have not yet had a PageRank moment for intelligence.

—Adrian Kosowski

In the 1990s, web search already existed: we could crawl and index the web, and it was growing very quickly. Then the PageRank algorithm was developed.

It introduced a mathematical idea: to model the web as a graph and compute the stationary distribution of a random walk.

Then, Google combined it with the systems needed to compute it at internet scale.

This gave search a clear center: a well-defined mathematical object (stationary distribution, web graph, etc.) to compute.

For transformers, Kosowski argues we do not yet have an equivalent. We are given some useful tools (attention, next-token prediction, and scaling), but still lack a widely accepted mathematical object that captures the core computation behind intelligence.

---

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu (AI Developer, BTech, [LinkedIn](https://www.linkedin.com/in/sai-krishna-pashapu-997358283/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained), [Github](https://github.com/saikrishna-pashapu?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained))
- Bob Boncella
- Demet Yazici
- Imran Khan

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 8000+ ML engineers? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=the-dragon-hatchling-bdh-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team