# Diffusion Models, Simply Explained

#57 Jun 30, 2025 Architecture

https://www.aibutsimple.com/p/diffusion-models-simply-explained

Diffusion models are generative deep learning models that have skyrocketed in popularity in recent years due to their exceptional performance, beating out other established generative models like the [Generative Adversarial Network (GAN)](https://www.aibutsimple.com/p/generative-adversarial-networks-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained)and [Variational Autoencoder (VAE)](https://www.aibutsimple.com/p/a-visual-guide-to-variational-autoencoders-vaes?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained).

Diffusion models are most known for their image generation abilities—they are used in popular generative models such as DALL-E or Stable Diffusion.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2292e693-02c5-4095-b744-481134204cf4/Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_800_px___4_.png)

Even though diffusion models were introduced in 2015, they didn’t start taking off until early 2020, when [Ho et al. released a paper](https://arxiv.org/abs/2006.11239?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained) proving how probabilistic diffusion models could be effectively applied to image synthesis.

That’s great, but how do they work, and what even is diffusion? Diffusion refers to the phenomenon of a “scattering” of particles, and it is widely studied in the field of thermodynamics.

An example of this is sunshine that passes through clouds, creating rays that scatter and spread out in different directions. Diffusion is occurring from the random motion of the light particles.

This natural process is similar to what happens in diffusion models used for image generation. Random noise is added to an input image, progressively showing scattered pixels of colors that differ from the original image.

Specifically, diffusion models add Gaussian noise to training data successively, which we refer to as the forward diffusion process, then learn to reconstruct the data by reversing this noising process—which we refer to as the reverse diffusion process.

- Essentially, diffusion models add noise to images, then learn to remove the noise they added.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8011487c-5842-4dd7-bbf3-c2072c2388ee/Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_400_px___1_.png)

After training, diffusion models will be able to generate high-quality images starting from just random noise inputs, iteratively denoising them to create full images of anything the model was trained on.

---

### Stop Asking AI Questions, and Start Building Personal AI Software.

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b3228f7e-0681-4ba7-b87f-76a9e9440681/AI_Fast_Track_Header__3_.png)](https://www.alliekmiller.com/the-ai-fast-track-confirmation)

Transform your AI skills in just 5 days through this [free email course](https://www.alliekmiller.com/the-ai-fast-track-confirmation). Whatever your starting point, by Day 5 you'll be building working software without writing code.

Each day delivers actionable techniques and real-world examples straight to your inbox. No technical skills required, just knowledge you can apply immediately.

[Sign up for the course](https://www.alliekmiller.com/the-ai-fast-track-confirmation)

---

### Diffusion Model Architecture

So we know that diffusion models have a forward and reverse diffusion process, but what neural architecture supports this process?

In the forward diffusion process, or the noising process, no neural network architecture is involved. Gaussian noise is added to input data over several time steps by sampling from a normal distribution. This process can be defined in one math equation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/447744b5-fdaf-4995-abe9-9960594abedc/Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_800_px_.png)

On the other hand, in the reverse diffusion process, where all the denoising occurs, U-Net architectures are commonly used. [U-Nets (Ronneberger et al., 2015)](https://arxiv.org/abs/1505.04597?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained) are convolutional neural networks (CNNs) that were developed for image segmentation in the biomedical space.

Here, however, U-Nets are not used for that same purpose. Convolutional layers are helpful for diffusion, but primarily, it is the unique “U-shaped” architecture of the U-Net that gives it an advantage in diffusion.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fa40de39-b579-4f99-b0f8-6a204a8bb6f3/image.png)

Sample U-Net architecture from the original paper.

The only real restriction for the architecture of diffusion models is that input and output image sizes should be the same—a consistent size allows the denoising process to flow smoothly and predict noise at a pixel level.

For this reason, it makes a lot of sense for the U-Net to be popular. One of the primary characteristics of the U-Net is that it’s able to preserve the input and output image dimensions throughout the diffusion process.

Additionally, the U-Net's encoder-decoder architecture with residual connections makes it well-suited for denoising. Image features are directly transferred through the residual connections.

### Forward Diffusion

The forward diffusion process is deterministic and non-learnable, meaning that the noise is applied predictably and the same every time, and no model with parameters is involved.

This procedure gradually transforms a clean input image into pure noise after a certain number of timesteps. We can denote this series of discrete timesteps as T.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/71242c70-0a4e-423f-9303-590a17dfed6a/image.png)

Here, q (dashed line) represents the forward process.

Starting with the initial clean data point (x0), we incrementally add Gaussian noise following an equation that represents something called a Markov chain.

- A Markov chain is a way to describe a random process where you move from one state (or situation) to another. Each possible move has a fixed probability.
- At each step, where you go next depends only on where you are now, not where you were before.

Mathematically, the noisy data xt at timestep t is sampled from a Gaussian distribution conditioned on the data from the previous step, xt−1.

Each step of the forward diffusion process is defined by this equation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/054c1573-6e46-400c-abdb-53fe00b8923f/image.png)

Here, q is our forward process, xt​ is the output of the forward process, and N is a normal distribution with sqrt(1−βt)xt-1​ as the mean and βt​I as the variance.

There’s another factor (and hyperparameter) that we must consider during our noising process. We call it the noise schedule (βt), and it represents how much noise we apply at each timestep along with the rate at which this applied noise increases or decreases.

For instance, linear noise schedules apply noise at a constant rate across timesteps, and they are quite popular—although not that efficient. Non-linear noise schedules are also found, including cosine or sigmoid schedules.

### Reverse Diffusion

The reverse process, or the denoising process, is where pure noise is transformed back into an image using a U-Net architecture. This part of the diffusion process is where the image is “generated” from noise.

The main goal of the reverse diffusion process is to learn the noise that was added during the forward process. If we learn this noise, we are able to recover the original data through an iterative denoising process.

In the paper, the authors describe modeling this process using a parameterized reverse Markov chain, where a neural network—hence, parameterized—is trained to approximate the added noise of the whole image q.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/539b7508-d346-4bdb-acc9-71186e9f590d/image.png)

Here, p is the parameterized approximation, and it is represented using a normal distribution with a mean of μθ​(xt, t), predicted by a neural network, and a variance of ∑θ(xt,t), which is often fixed for simplicity to the original βt​I.

A common misconception is that the model attempts to predict the noise to reach the next timestep t-1. However, the model is actually trying to predict the entire noise component that is required to return to the original image (t=0) at each step.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/55ef326c-1f8c-4078-a12f-fd4de7a97c8d/Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_800_px___1080_x_600_px___1_.png)

Afterwards, a part of this noise component is subtracted from the input image based on the noise schedule and fed as the next input.

Basically, the reverse diffusion process involves iteratively applying these following steps. Start with random noise xT​, and at each step t (from T to 1), use the trained neural network ϵθ​ to predict the noise component in xt​.

Then, use this predicted noise to calculate the mean μθ​(xt​,t) and sample xt−1​ from a Gaussian distribution with that mean and a fixed variance. This gradually removes noise until a clean image x0​ is generated.

### Training Algorithm

The training algorithm is quite short and simple; here is the entire pseudocode from the paper:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b8bd6976-5a17-4e9d-9e02-6e523ce091d9/image.png)

First, we sample an image from our dataset (step 2). Then, we sample a random timestep t (step 3) and sample noise ϵ from the normal distribution (step 4).

We noise our selected image with the sample Gaussian noise ϵ in the forward diffusion process—we can apply it without going through the iterative process by manipulating the math equation for q.

- We go through more of the math of the diffusion process for supporters, so consider [upgrading](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained)to access better math explanations.

The generated noisy version of the image is fed into the reverse diffusion process, using a U-Net to predict the total noise component present in the image.

The model’s performance is then evaluated by calculating the loss between the predicted noise and the actual noise (ϵ-ϵθ)2. We minimize this loss using gradient descent, then repeat the entire process until the model converges.

### Sampling & Generation

Once the model has been thoroughly trained with the forward and reverse process, the sampling process for generating new data involves performing the reverse process from image samples of pure Gaussian noise.

The model then iteratively applies denoising steps using the trained neural network. Noise is gradually transformed into a newly generated image.

To prevent the generation process from becoming overly deterministic and generating the same images every time, a small amount of noise is usually added at each timestep during sampling.

### Comparison

Diffusion models for image generation have blown past previous SOTA models like GANs, VAEs, flow-based models, and autoregressive models by achieving high-quality generation with the lowest loss.

Here’s a quick comparison of some of the most popular generative model architectures:

- Diffusion Models generate data by simulating a reverse diffusion process. They initiate from a random noise distribution and gradually remove noise through iterative steps. The model attempts to predict and remove the total noise at each step.
- Generative Adversarial Networks (GANs) perform adversarial training, involving two competing neural networks—a generator and a discriminator. The generator tries to fool the discriminator by generating data indistinguishable from real data, while the discriminator's task is to differentiate between real and generated data.
- Variational Autoencoders (VAEs) learn the distribution of the input data while encoding data into a latent space, then sample from the distribution to generate images. Like other autoencoders, they are made up of an encoder and decoder, where the encoder compresses data into a latent space and the decoder reconstructs data back into its original form.

Diffusion models should be chosen for the generation of high-quality and diverse outputs, training stability, and generating data from complex multi-dimensional image distributions.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=diffusion-models-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team