Issue #105 Architecture Edwin Dong
The Dragon Hatchling (BDH), Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook, diving into the BDH-GPU architecture in detail with intuitive Python code.
We’ll train it on a byte version of the Tiny Shakespeare, and compare depths by ablation. We’ll investigate the sparsity of the ReLU block, explore the activations, and inference our very own architecture. Please do read the full article before attempting the code tutorial.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank can be found here. This month, we’ve incorporated more company-specific interview questions.
These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic.
Check back frequently for more questions, and let us know what type of questions you would like to see next!
For nearly a decade, the Transformer architecture has been the undisputed frontrunner of modern deep learning.
However, as models scale into the trillions of parameters, the quadratic scaling O(n2 ) of the self-attention mechanism is a harsh limitation.
As a model’s context window grows, the Key-Value (KV) cache memory requirements explode, making an ideal "infinite context," where a model continuously learns like our brains do, practically impossible on standard hardware.
Additionally, recent complexity theory proofs suggest that autoregressive attention is mathematically bound to hallucinate at predictable rates.
In response, recent research has seen a pivot toward "Post-Transformer" architectures.
These models prioritize linear-time inference, constant memory bounds, dynamic state evolution, and the ability to learn continuously without forgetting.
In the post-transformer field, we’ve seen the emergence of State-Space Models, and linear transformer backbones such as the Gated DeltaNet, among other architectures.
One specific post-transformer architecture sticks out from the rest. Interestingly, The Dragon Hatchling (BDH) (Kosowski et al., 2025) models its architecture after the human brain.
Created by a team of researchers at Pathway (including ex-Google Brain, Inria, and Mila), the BDH is promising for long-horizon reasoning and is a strong candidate for continual learning.
This edition is for PRO members.
