# Diffusion Transformers (DiTs), Simply Explained

#44 Mar 31, 2025 Architecture

https://www.aibutsimple.com/p/diffusion-transformers-dits-simply-explained-002a

The diffusion transformer (DiT), proposed by William Peebles at UC Berkeley and Saining Xie at New York University, is a type of generative AI model that blends two powerful machine learning techniques: diffusion models and transformers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1bf755d5-3701-4bac-9882-4e6f980f3c0b/image.png)

DiT Architecture

The original paper, “[Scalable Diffusion Models with Transformers](https://arxiv.org/abs/2212.09748?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)”, was released in 2022. Feel free to check it out on arXiv!

### What is diffusion?

Diffusion refers to the phenomenon of a “scattering” of particles. An example of this is sunshine that passes through clouds, creating rays that scatter and spread out in different directions. Diffusion is occurring from the random motion of the light particles.

This is similar to what happens in diffusion models ([Ho et al., 2020](https://arxiv.org/abs/2006.11239?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)) used for image generation. Random noise is added to the image, progressively displaying scattered pixels of colors that deviate from the original image.

Specifically, diffusion models add Gaussian noise to training data successively (forward diffusion process), then learn to reconstruct the data by reversing this noising process (reverse diffusion process).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/31177be1-a728-4be6-8f1e-25c9d745d1ed/Copy_of_Copy_of_noisy_latnet__720_x_400_px___1_.png)

After training, these models can generate high-quality images starting from random noise—iteratively removing it to create full images of just about anything it was trained on.

### What are transformers?

A transformer model ([Vaswani, et al., 2017](https://arxiv.org/abs/1706.03762?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)) is a type of neural network architecture that has become the foundation for many modern natural language processing (NLP) tasks, such as chatbots, translation, and text generation. It is the core architecture behind GPT-3.

The main innovation behind the transformer is the self-attention mechanism, which was originally used for machine translation in RNNs. This mechanism is extremely useful, allowing the model to learn global relationships in the input data.

The attention mechanism allows transformers to weigh the importance of different tokens (words or symbols) in an input sequence when making predictions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/94f1c885-3545-41bf-b9de-d4f5910f529d/cropped_27cc62e6-1090-4113-ae77-a5f9ac132f7d_1743275979511.png)

The Transformer Architecture

If you aren’t familiar with transformers, or if you just want a deep dive (including all the math), you can check out our past issues detailing them here:

- [Transformers and Multi-head Attention, Step-by-Step (Encoder)](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)
- [Transformers and Multi-head Attention, Step-by-Step (Decoder)](https://www.aibutsimple.com/p/transformers-and-self-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)

---

Big thanks to our partners for keeping this newsletter free. If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you’ll enjoy.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ff79e3ad-93d4-4cc0-8540-6a3b58d70541/Ad_The_AI_report.png)

### There’s a reason 400,000 professionals read this daily.

Join The AI Report, trusted by 400,000+ professionals at Google, Microsoft, and OpenAI. Get daily insights, tools, and strategies to master practical AI skills that drive results.

Sign up now for free and work smarter, not harder.

---

### How Do DiTs Combine The Two?

As mentioned previously, diffusion models operate on a two-stage process: the forward diffusion process and the reverse diffusion process.

During the reverse diffusion process, the denoising is usually done by a U-Net, a relatively powerful convolutional neural network (CNN) that was developed for image segmentation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5f13cc16-b180-47a6-8108-5e70371768da/image.png)

Sample U-Net Architecture

The DiT takes this commonly used U-Net backbone in diffusion models and replaces it with a transformer architecture.

Now, let’s take a closer look at how the DiT process works. First, the input layer receives conditional information, providing useful context for the generative process.

Then, an input image is compressed into a latent space (a lower-dimensional space) using an autoencoder, which learns to capture essential features. We covered the autoencoder in a previous issue; feel free to check it out below:

- [Introduction to Autoencoders (Beginner)](https://www.aibutsimple.com/p/deep-unsupervised-learning-autoencoders?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/da279362-99c4-414c-860e-6363cb5e7bc3/Untitled_design__37_.png)

The image’s latent representation is then divided into square patches, treated as tokens, and subjected to the forward diffusion process, where noise is gradually added. This portion of the model is called the diffusion layer.

The transformer backbone, using techniques such as adaptive layer normalization (adaLN), performs the backwards diffusion process, processing the noisy latent tokens, predicting and removing the noise iteratively (over timesteps).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2e362ce0-5735-401c-8978-a97f51df6d7a/Transformer_Backbone.png)

Finally, the denoised latent representation is mapped back into an image. This is typically done by another autoencoder—typically a variational autoencoder (VAE).

The DiT denoising process leverages a transformer’s ability to handle global context through self-attention, potentially improving the resulting image’s quality. Using transformers instead of the U-Net allows for better scalability and similar or better performance.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e11a7257-43be-4c5d-85ee-2a299bc91ad7/Transformer_Backbone__1_.png)

### Worked Example

Let’s walk through a straightforward example with some math.

To illustrate how data flows through a sample DiT architecture, consider a simplified example with a transformer embedding dimension of d = 2 and single-head attention.

#### Latent Representation and Noise

To start, keep in mind that during the forward diffusion process, noise is added progressively to the latent representation over T steps.

Assume we have some autoencoder that extracts essential features from an input image and gives us a 2×2 latent representation (z0) with these values:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1a6e96c3-bfdc-4d48-b830-6574e0f71af2/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add___3_.png)

This is a small, simplified version of our image, where each number represents a part of the image.

We also define a noise matrix ϵ with some sample values and a parameter āt that decreases from 1 to 0 as timestep t increases from 1 to T, controlling the amount of noise added.

We compute the noisy latent for some random timestep t where āt is 0.9 through this formula:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e9474f5c-9cc4-4b48-aaca-c7d726366feb/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy__1_.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/983e834f-5e5e-4280-bdfd-18de3f11761d/noisy_latnet__2_.png)

Think of this noising as adding a bit of random fuzz to our compressed image, making it grainy. Keep in mind that the numbers may not add up due to rounding to three decimal places.

In addition to noised image inputs, diffusion models sometimes process additional conditional information such as noise timesteps t, class labels c, natural language, etc. However, for simplicity purposes, we will skip those computations.

#### Tokenize, Embed, and Position

Now, we’ll treat each cell of the noised latent as a token (so we have 4 tokens: -0.917, 0.063, 1.044, and 2.024).

The transformer needs vectors, not single numbers, so we embed each token into a 2D vector. For simplicity, let’s repeat the number:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6a46e9a2-d8db-4c12-a73d-246ebde1ad18/Copy_of_noisy_latnet__1_.png)

After embedding, we’ll positionally encode them, like any other transformer. We’ll add the token embeddings to arbitrarily chosen positional embeddings (not the sinusoidal ones in the transformer paper) and stack them into a matrix X.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c7acbe3d-7f03-4b5d-83c6-2577776c310d/Copy_of_Copy_of_noisy_latnet.png)

Now, the transformer knows the position of each token.

#### Self-Attention Mechanism

The transformer uses self-attention, which helps it learn global dependencies and weigh the importance of image patches. For each pair of image patches, a score is computed between them.

Let’s assume one head (single-headed attention) and use some random query, key, and value weight matrices. Here are our queries (Q), keys (K), and values (V):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a8d76831-b22b-4268-a17a-ff956dad54a8/Copy_of_Copy_of_Copy_of_noisy_latnet.png)

Now, we’ll compute the attention scores by performing a matrix multiplication of the query matrix and the key matrix transpose, then scaling it by the square root of the embedding dimension (d = 2).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/826499c8-347d-4b1e-80c8-4058c5c04be7/Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet.png)

We apply the softmax activation function to these scores to get weights (exponential and normalize), then multiply the softmax output with our value matrix to obtain the output of our simplified transformer portion.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/71467d78-72d0-440e-aa6e-1185a58e5449/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__2_.png)

At this point, we can perform layer normalization and all the other fancy manipulations the paper recommends (seen in the adaLN-Zero version), but those require large amounts of additional computations—we’ll skip those for simplicity purposes.

#### Predict Noise and Denoise

To keep things simple, we’ll use the attention output matrix as the final noise prediction to start the denoising process.

We reshape the attention output back to the original 2×2 latent shape (the tokens originally came from a 2×2 grid) by taking the first 2 rows of the matrix.

- In a full DiT model, the transformer output would undergo a feedforward network and many more transformations, finally being projected back into the latent space in the right dimensions (2×2).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/744769f8-d529-433e-927d-ba82619ca147/Copy_of_noisy_latnet__4_.png)

Denoising involves using the predicted noise ϵ^ to compute a less noisy latent zt-1​. DiT often uses the DDIM (Denoising Diffusion Implicit Models) update rule for efficient sampling, which lets the model skip timesteps while maintaining quality. The DDIM update rule is this huge equation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b51ad76f-ee91-48c4-a3b1-5715e3ce5ee6/image.png)

Now, let’s compute zt-1 using our noisy latent zt and our noise prediction ϵ^ .

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2127eb87-118c-43a6-9783-d9c37a871709/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_450_px_.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f2601943-0d25-4f35-80de-13ffc169d4c1/Copy_of_Copy_of_noisy_latnet__2_.png)

But the process doesn’t end there. Denoising, just like adding noise, is an iterative process that will require hundreds or thousands of iterations, slowly removing noise until we reach the final latent image z0. This is referred to as the clean latent.

Finally, since we’re still in the latent space, we take the final denoised image z0 and decode it back to the pixel space, typically using a variational autoencoder (VAE).

The output from the VAE will be our final generated image, which can be compared to our input image to measure a loss to evaluate the quality of the generated image.

### Performance

The largest DiT-XL/2 models outperformed all prior diffusion models on the Imagenet dataset. It achieved a Fréchet Inception Distance (FID) of 2.27 for 256x256 images, a state-of-the-art performance in 2022.

- FID is a widely used metric to evaluate the quality of generated images by comparing their feature distributions to those of real images.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b990f487-e8d9-43e6-bf2b-416791aea677/image.png)

In terms of scalability, larger DiT models, with increased transformer depth and width or more input tokens, consistently lowered the FID, suggesting that DiT inherits the scaling properties of transformers seen in NLP.

- The paper mentions, “DiT Gflops are critical to improving performance.”

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-transformers-dits-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team