# Energy-Based Transformers (EBTs), Simply Explained #77 Nov 17, 2025 Architecture https://www.aibutsimple.com/p/energy-based-transformers-ebts-simply-explained Recent advances in deep learning have brought us models that can generate text, solve code, and describe images with great accuracy and fluency. Yet, these models, as powerful as they are, remain mostly System 1 thinkers: fast, pattern-recognizing models that excel at recall and imitation. They still struggle with slow, deliberate reasoning, known in cognitive science as System 2 Thinking. The paper “[Energy-Based Transformers are Scalable Learners and Thinkers](https://arxiv.org/abs/2507.02092?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained)” by Gladstone et al. introduces a new technique to solve this issue by merging two previously separate ideas: Energy-Based Models (EBMs) and Transformers. The result is Energy-Based Transformers (EBTs), which are models that not only learn from data efficiently but can also think at inference time by iteratively refining their predictions. - They propose a new way for transformers to “reason”, suggesting an alternative to current reasoning solutions such as chain-of-thought (CoT) reasoning. In simple terms, EBTs can turn Transformers from pattern recognizers into problem solvers, capable of verifying their own reasoning before answering. --- ### An AI scheduling assistant that lives up to the hype. [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/39bb7246-a8f9-42c5-83c7-5281f81e63d7/skej_primary1.jpg)](https://skej.com/?utm_source=beehiiv&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_45910909-c977-4a55-b765-fe7a49b3c11f_07437066) [Skej](https://skej.com/?utm_source=beehiiv&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_45910909-c977-4a55-b765-fe7a49b3c11f_07437066) is an AI scheduling assistant that works just like a human. You can CC Skej on any email, and watch it book all your meetings. Skej handles scheduling, rescheduling, and event reminders. Imagine life with a 24/7 assistant who responds so naturally, you’ll forget it’s AI. [Free to try at Skej.com](https://skej.com/?utm_source=beehiiv&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_45910909-c977-4a55-b765-fe7a49b3c11f_07437066) --- ### The Motivation: From Generating to Verifying Conventional Transformers like GPT or BERT learn to map an input (x) directly to an output (y) using a parameterized function fθ: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f685775f-52e2-4b4d-b65a-7fbf8d882a39/image.png) They predict all tokens in parallel (BERT) or sequentially (GPT) using attention layers and feed-forward blocks. Once they are trained, each forward pass produces one output. However, humans rarely operate that way. We verify our thoughts; if an answer seems uncertain, we mentally iterate until we’re satisfied. This verification ability is central to System 2 Thinking: slow, analytical reasoning guided by self-evaluation. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d28e5804-9478-45d9-86e0-17d327761fc2/System_1_vs_System_2_Energy_Based_Transformers__5_.png) EBTs bring this idea into neural networks by reframing prediction not as a direct mapping but as an energy-minimization process. The model learns an energy function Eθ: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5daa441c-e333-4b24-ad8d-5c85138e7740/image.png) The energy function scores how compatible a prediction y is with its context x. Low energy means more compatible, which makes the output a better answer. At inference time, EBTs don’t just output an answer. Instead, they search for the lowest-energy prediction, refining it through gradient descent. In essence, they think by optimization. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0fb0d39b-2a6c-47fd-8f26-ea2a6e5ac74f/Energy_Graph_EBT__2_.png) ### Energy-Based Models Energy-Based Models assign a scalar energy value to every configuration of variables. For a probabilistic EBM, this defines a [Boltzmann-like](https://en.wikipedia.org/wiki/Boltzmann_distribution?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) distribution: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/122f46d1-3034-4e45-b5f1-dcebad0dd319/image.png) Here, (z) is a partition function integrating over all possible states. Because computing (z) is infeasible in high dimensions, modern EBMs usually ignore normalization and work with unnormalized energies. Training the models is conceptually simple: - Good (real) samples should have low energy. - Bad (fake) samples should have high energy. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c9bc76b0-280f-401c-baa2-0fb9a61ed9db/Energy_Function_EBT__1_.png) This energy landscape becomes a map of what the model considers “likely” or “valid.” In supervised or self-supervised setups (like language modeling), the model minimizes energy for the correct output (y) given the context (x). ### From Energy-Based Models to Transformers Traditional EBMs have long been difficult to scale, and they require contrastive learning or [Markov Chain Monte Carlo](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) sampling that becomes impractical in large dimensions. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c81aee18-29f0-4ca8-8257-4ad3bfe94389/Input__Traditional_Transformer__2_.png) Energy-Based Transformers (EBTs) solve this by embedding the energy function inside a Transformer architecture, leveraging its strength and stability. The Transformer backbone provides representational power, while the energy-based formulation adds the reasoning capability. There are two main EBT variants: 1. Decoder-only EBT (like GPT), which is autoregressive, predicting the next token through causal attention. 2. Bidirectional EBT (like BERT), which allows masked prediction and infilling using bidirectional context. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/36448fe5-d8df-4397-b324-6c21c36f8de9/Energy_Based_Transformers__EBTs_.png) During both training and inference, these models operate not by direct prediction, but through energy optimization. They iteratively update candidate outputs to reduce their energy. ### Learning to Verify: Easier Than Generating A core idea behind EBTs is that verifying a candidate solution is easier than generating one from scratch. This verification process is similar to the discriminator in [generative adversarial networks (GANs)](https://www.aibutsimple.com/p/generative-adversarial-networks-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained). ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fa5f10f8-602e-40df-bd2f-237574fb61f3/Generation_EBTs__1_.png) In [complexity theory](https://en.wikipedia.org/wiki/Computational_complexity_theory?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained), verifying a path through a maze is easier than discovering it. Similarly, an EBT learns to verify the correctness of a candidate prediction rather than memorizing all the possible answers. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ff3dc53-f3d9-4852-8073-20316c545d8a/Verification__simpler__EBT__3_.png) This verification-driven approach offers a natural advantage: - It generalizes better to unseen data (since verification logic often transfers). - It avoids instability from adversarial training common in generator-discriminator setups like GANs. - It unifies generation and evaluation inside a single model, which means the generator is just the gradient of the verifier. ### The “Thinking” Process: Optimization as Reasoning When an EBT receives an input context (x), it begins with a random or rough candidate prediction (y0). Then, it iteratively updates this prediction using the gradient (∇) of the energy function: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bc9740c6-8ec4-46db-b2c8-08817afb76e1/image.png) Each step moves the prediction toward regions of lower energy, which are the regions the model deems more coherent with the context. This process mimics a reasoning loop: each gradient step is like a “thought,” refining the answer until convergence. When the energy stops decreasing, the model “decides” it has thought enough—achieving a form of adaptive computation during inference. The authors visualize this as an energy landscape, where the model descends toward the lowest valley representing the best answer. Complex or ambiguous inputs yield rougher landscapes, meaning the model must take more steps (it thinks longer), just like humans do with harder problems. ### System 1 vs. System 2 EBTs The paper defines two operational modes: - System 1 (EBT-S1): quick, feed-forward predictions with detached gradients between steps for efficiency and stability. - System 2 (EBT-S2): full iterative optimization without gradient detachment, allowing deeper reasoning through multiple refinement steps. Interestingly, S2 models may train slightly slower but scale faster in performance as they grow, mirroring human cognition: slow to think, powerful once matured. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) --- ### Scaling Results Across extensive experiments on both discrete (language) and continuous (vision) tasks, EBTs exhibit superior [scaling laws](https://www.aibutsimple.com/p/neural-scaling-laws-simply-explained-16c1?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) compared to standard Transformer models. #### During Training (Learning Scalability) EBTs achieved up to a 35% higher scaling rate across parameters, depth, batch size, and compute. That means they learn more efficiently as they grow and need fewer tokens or compute to reach the same loss. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/dbbc0c4f-22fa-4d6f-b17b-ae5b3f77b32e/Untitled_design__7_.png) #### During Inference (Thinking Scalability) When allowed extra “thinking steps,” EBTs improved performance 29% more than Transformers on language modeling and outperformed Diffusion Transformers (DiTs) on image denoising using fewer forward passes. Moreover, the gains are larger on out-of-distribution data, suggesting EBTs can reason their way through unfamiliar situations rather than overfit training distributions. ### Comparison with Diffusion Models Diffusion models, like those used in image generation, also iteratively refine predictions. However, they predict noise, not energy. As a result: - Diffusion models can’t directly compare how “good” two intermediate states are; they lack explicit compatibility scores. - Their denoising schedule is fixed, so computation can’t adapt dynamically. EBTs generalize this process. Each step produces an unnormalized likelihood estimate, letting the model decide when to stop refining. This flexibility makes EBTs a more natural framework for reasoning and adaptive computation. ### Energy Landscape Regularization: Making Thinking Stable A challenge in training EBMs is ensuring that the learned energy landscape isn’t chaotic. Too many local minima can trap the model’s optimization. To address this, EBTs introduce energy landscape regularization techniques that smooth and make the landscape more convex (concave up). - This leads to more predictable optimization trajectories and, remarkably, enables System 2 Thinking to emerge naturally during pretraining. In other words, EBTs don’t need explicit reasoning supervision; the ability to “think” arises as a byproduct of learning a smooth energy landscape created with regularization techniques. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b7adf74f-0ce1-4d26-82f3-b79e9a918230/Regularization_for_EBTs.png) ### Simple Example: Thinking as Energy Descent Imagine predicting the next word in a sentence. A standard Transformer picks one directly, while EBTs start with a random guess and refine that guess step by step: | Step | Candidate | Energy | Interpretation | | 0 | random noise | high | nonsensical guess | | 3 | “fruit” | medium | partially coherent | | 5 | “apple” | low | contextually correct | Each energy value represents how compatible the word is with the context of the sentence. This iterative refinement visualizes thinking—the model learns when it has reached a good answer because the energy then stops decreasing. ### Limitations and Future Outlook The authors acknowledge key limitations in their paper: - EBTs can require tedious tuning of optimization hyperparameters (step size α, number of steps N, etc.). - Training is more computationally expensive due to gradient-based inference. - Large-scale experiments (beyond 800M parameters) remain unexplored due to resource constraints. - Multimodal generation still poses challenges due to the convex-landscape assumption—the actual landscape can be rougher. Yet, the results suggest that as compute grows, EBTs’ efficiency advantage will dominate, making them promising candidates for future [foundation-model architectures](https://aws.amazon.com/what-is/foundation-models/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained). ### Conclusion Energy-Based Transformers mark a turning point in the evolution of intelligent systems: they bridge the gap between fast pattern recognition and deliberate reasoning, teaching models not just to generate outputs but to understand and verify them. From an engineering perspective, EBTs introduce a scalable, unified framework for learning and thinking; from a cognitive perspective, they move AI one step closer to human-like reasoning. As future research explores larger models and richer energy landscapes, EBTs may well redefine how we design models that not only learn faster but also think better. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=energy-based-transformers-ebts-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team