Diffusion Language Models (DLMs), Simply Explained

Many language models you use today work autoregressively, generating tokens left to right given the context of what came before.

LLMs like GPT, LLaMA, and Claude all mostly follow this structure.

This idea is so deeply baked into how we build these models that most researchers also treat it as a given.

A team from Renmin University and Ant Group decided to ask a simple question: Does it always have to be this way?

Can language models instead be guided by the stochastic process of diffusion for generating text, much like for generating high-fidelity images and video?

Their answer is in the form of a model called LLaDA (Nie et al., 2025), and it is a new interesting direction.

What You’ll Learn

1. The reversal curse (the consequence of autoregressive reasoning)

2. How the LLaDA masked diffusion model works

3. Why LLaDA opts for bidirectional attention

4. LLaDA results and benchmarks and what they show

5. DiffusionGemma, Google’s new diffusion language model

6. Flow matching and embedding diffusion

What's Helpful to Know

• Autoregressive models (ARM)

  • Models that generate tokens one at a time (left to right), where each token is conditioned on all previous tokens. This is how most current LLMs work.

  • More precisely, they factorize a joint distribution into a product of conditional distributions.

    

• Diffusion models

  • Originally popularized for images, they are generative models that learn to reverse a noising process. Data is iteratively corrupted, then a diffusion model learns to reverse this corruption.

  • For images, this means adding Gaussian noise and learning to remove it. For text, they use a discrete corruption process over tokens.

• Masked diffusion model (MDM)

  • A diffusion model that corrupts text by randomly replacing tokens with [MASK] tokens, then trains a neural network to predict what the masked tokens originally were.

When Autoregressive Models Break Down

Before getting into LLaDA, it helps to understand with a simple example the one specific place where the left-to-right approach in autoregressive models can break down.

When you ask GPT-4 to complete the next line of a poem, it does fine, but if you ask it to complete the previous line, performance drops sharply.

This is not because the model lacks knowledge of poetry, but instead because the model was trained to predict what comes after and not what came before.

Essentially, it has no mechanism to sharply reason in the other direction.