NRGPT: Combining Energy-Based Models With GPT

In partnership with Section:

Transformers for autoregressive language modelling, such as GPT, are typically understood as deep feed‑forward sequence models trained to maximize the probability of correct tokens, which we call maximum likelihood estimation (MLE) in statistics.

While this view has led to extraordinary success, researchers have been exploring new paths for transformer inference, reasoning, optimization, and dynamical systems.

The NRGPT paper (NRGPT: An Energy-based Alternative for GPT https://arxiv.org/abs/2512.16762v1) proposes an energy‑based version of autoregressive Transformers.

• They show that the forward pass of GPT can be interpreted as iterative gradient descent on a learned energy function.

But what is an energy function, and what is energy? How can this be extended to transformers like GPT, and how does it help them reason and choose better answers?

In this issue, we’ll dive deep into NRGPT and the underlying math, categorize it within the broader family of Energy‑Based Models (EBMs), and clarify how it differs from Energy‑Based Transformers (EBTs). 

We will see how NRGPT transforms GPT inference into a stable dynamical system and why this perspective offers a new approach to generalization and reasoning behavior.