Energy-Based Transformers (EBTs), Simply Explained

Recent advances in deep learning have brought us models that can generate text, solve code, and describe images with great accuracy and fluency.

Yet, these models, as powerful as they are, remain mostly System 1 thinkers: fast, pattern-recognizing models that excel at recall and imitation. They still struggle with slow, deliberate reasoning, known in cognitive science as System 2 Thinking.

The paper “Energy-Based Transformers are Scalable Learners and Thinkers” by Gladstone et al. introduces a new technique to solve this issue by merging two previously separate ideas: Energy-Based Models (EBMs) and Transformers.