Issue #77 Architecture Edwin Dong, Anurag Shinde
Energy-Based Transformers (EBTs), Simply Explained
Free editionPRO edition
This week’s PRO issue is accompanied by a Colab notebook filled with amazing explanations and LaTeX equations for a simple EBT example. As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Recent advances in deep learning have brought us models that can generate text, solve code, and describe images with great accuracy and fluency.
Yet, these models, as powerful as they are, remain mostly System 1 thinkers: fast, pattern-recognizing models that excel at recall and imitation. They still struggle with slow, deliberate reasoning, known in cognitive science as System 2 Thinking.
The paper “Energy-Based Transformers are Scalable Learners and Thinkers” by Gladstone et al. introduces a new technique to solve this issue by merging two previously separate ideas: Energy-Based Models (EBMs) and Transformers.
The result is Energy-Based Transformers (EBTs), which are models that not only learn from data efficiently but can also think at inference time by iteratively refining their predictions.
They propose a new way for transformers to “reason”, suggesting an alternative to current reasoning solutions such as chain-of-thought (CoT) reasoning.
In simple terms, EBTs can turn Transformers from pattern recognizers into problem solvers, capable of verifying their own reasoning before answering.
This edition is for PRO members.
