# A Visual Guide To Variational Autoencoders (VAEs)

#45 Apr 7, 2025 Architecture

https://www.aibutsimple.com/p/a-visual-guide-to-variational-autoencoders-vaes

In Partnership With:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3b8ff4dc-bc3b-4be6-ab6b-da2105547ac4/image.png)

Variational autoencoders (VAEs) are autoencoder-based neural networks used for generative tasks such as image generation or reconstructing missing data.

They are comparable to traditional autoencoders but use probabilistic techniques to not only reconstruct input data but also generate new, similar data to what they were trained on.

The most popular use cases for VAEs are:

- Generating new images, such as faces or digits.
- Anomaly detection by identifying data with high reconstruction error.
- Dimensionality reduction, reducing data dimensions for visualization.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4a453763-2eba-43b4-a3d9-60ebf7fd1cfa/image.png)

VAE-generated handwritten digits

Being a generative model, its performance is compared to other similar generative models, such as GANs or diffusion models. Want to learn more about GANs? We did a deep dive in one of our previous issues:

- [Generative Adversarial Networks, Simply Explained (Intermediate)](https://www.aibutsimple.com/p/generative-adversarial-networks-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-visual-guide-to-variational-autoencoders-vaes)

At its heart, a VAE still has the same structure and components as a standard autoencoder: it has both an encoder and a decoder.

Like standard autoencoders, VAEs map inputs into a latent space (a smaller, lower-dimensional representation), then reconstruct latent space vectors back to the input space.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/be7ee6fb-1fa9-4a59-8965-499ed93f7f5f/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___5_.png)

The key feature of VAEs is that, unlike regular autoencoders (which map inputs to fixed points in the latent space), VAEs learn a probability distribution over the data.

Given an input, the encoder maps it to a distribution in the latent space, outputting the parameters (mean and variance) of the distribution.

The decoder samples from this distribution back to the input space, reconstructing the data and generating new samples. Sampling from the learned distribution allows the model to obtain new but similar data points.

---

Big thanks to our partners for keeping this newsletter free. If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you’ll enjoy.

### The gold standard of business news

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/42dae68d-fc9f-4301-9ca6-be85353f38bd/Option_9.png)](https://morningbrew.com/daily/subscribe?utm_campaign=XIAEVJHTGJ&utm_medium=paid_newsletter&utm_source=beehiiv&_bhiiv=opp_cc610824-5d42-4e53-9b72-99225465e075_652e2efc)

[Morning Brew](https://morningbrew.com/daily/subscribe?utm_campaign=XIAEVJHTGJ&utm_medium=paid_newsletter&utm_source=beehiiv&_bhiiv=opp_cc610824-5d42-4e53-9b72-99225465e075_652e2efc) is transforming the way working professionals consume business news.

They skip the jargon and lengthy stories, and instead serve up the news impacting your life and career with a hint of wit and humor. This way, you’ll actually enjoy reading the news—and the information sticks.

Best part? [Morning Brew’s](https://morningbrew.com/daily/subscribe?utm_campaign=XIAEVJHTGJ&utm_medium=paid_newsletter&utm_source=beehiiv&_bhiiv=opp_cc610824-5d42-4e53-9b72-99225465e075_652e2efc) newsletter is completely free. Sign up in just 10 seconds and if you realize that you prefer long, dense, and boring business news—you can always go back to it.

[Join 4.3 Million Readers Now](https://morningbrew.com/daily/subscribe?utm_campaign=XIAEVJHTGJ&utm_medium=paid_newsletter&utm_source=beehiiv&_bhiiv=opp_cc610824-5d42-4e53-9b72-99225465e075_652e2efc)

---

### Why Not Use A Standard Autoencoder?

Standard autoencoders don’t have good generative abilities since the latent space is deterministic. This means for a given input, the encoder produces a fixed output. Every time you pass the same input, the same latent vector is produced.

When we sample from a regular autoencoder, since the encoder produces a single latent vector for each input, we would get the same result every time—there's no way to sample new latent points that correspond to new data.

On the other hand, VAEs, even for single inputs, can generate multiple outputs by sampling from a probability distribution.

Additionally, the smooth distribution makes sure that sampled points do not vary greatly from the input. This is also a double-edged sword, since the images produced from VAEs are typically very smooth and “blurry”.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4cd8dae8-636a-430f-a57a-18eb4240f4c8/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___4_.png)

### How Does The Math Work?

The process begins with an input data point x, which could be an image, some text, or other structured data.

#### Encoding

We pass this input data x to an encoder—which is usually a neural network—that maps the input to a probabilistic latent space.

The encoder outputs the parameters of a posterior distribution p(z|x) of the latent variable z given the data x, which is usually the mean μ and variance σ2 .

- The posterior distribution is the distribution of the latent variables z given an observed data point x. It tells us how the latent variables should be adjusted due to the observed data. This is the distribution we want the model to learn.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/49caa730-750b-498c-a0d8-81e8a530a66d/Untitled_design__43_.png)

However, the exact posterior p(z|x) is difficult to compute directly, so we use something called a variational approximation to approximate it. The approximate posterior distribution q(z|x), is typically modeled as a Gaussian distribution.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f15d5598-5f23-4e8d-afa0-fad6789abcb7/Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_400_px_.png)

The goal here is to make the approximate posterior distribution as close as possible to the exact posterior. We do this by optimizing the two parameters—the mean and the variance—that define the approximate distribution.

How can we optimize these parameters?

During training, we sample a point from this distribution to feed into the decoder. But we have to think about exactly how we sample. If we just sample from the distribution randomly, how is it possible to train this model? Can we even differentiate random sampling of a distribution?

To solve this problem, we use a technique known as the reparameterization trick, which makes the sampling process differentiable, giving us our gradients needed for backpropagation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2c0f605-9a04-4981-9db0-2a74e05135f8/Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_400_px___3_.png)

#### The Reparameterization Trick

The reparameterization trick allows us to use backpropagation to adjust the encoder’s internal parameters based on the difference between the predicted output and the actual output.

To illustrate how it works, let’s sample a random variable from the distribution:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/40f2fd7f-0946-4e51-918d-9d00f36c6efb/image.png)

To perform backpropagation (using the chain rule), we need to compute the gradient with respect to the sampled variable z, then calculate the gradients with respect to the model’s parameters (mean μ and variance σ2 ).

- However, calculating the gradient with respect to z (a randomly sampled variable) is mathematically undefined.

The reparameterization trick offers a nice solution. Instead of directly sampling z from its distribution N(μ,σ2), we decompose z into three different components: two determinate (fixed) and one stochastic component.

We outsource the random sampling to the stochastic component ϵ. This new variable (ϵ) serves as a source of randomness entirely independent of the model parameters.

Then, we reparametrize the latent variable. The final equation expressing z is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59481663-ad74-4d9b-92b6-4f4bcb3ab211/image.png)

Above, μ is the mean and σ is the standard deviation, and ϵ is a random sample from a normal distribution N(0,1).

- Note that for the reparameterization trick, we use the standard deviation instead of the variance, since we need to scale the standard normal variable ϵ by the standard deviation to match the distribution.

This function is fully differentiable; we can now calculate gradients with respect to μ and σ during backpropagation, allowing us to update our parameters. In reality, the model uses log-variance, so the variance can be converted to standard deviation through this formula.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a957e2a9-b0ba-47d9-a804-a1f552ae7401/image.png)

Intuitively, the randomness is outsourced to ϵ, which the encoder controls with μ and σ. The model can now "learn" how to adjust μ and σ (then σ2 ) to produce better z values.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f3abe5c7-1504-48c0-ac29-b909f6a8acdf/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_400_px___3_.png)

#### Decoding

The decoder is usually a neural network, and its job is to take latent variables (z) and transform them back into the original data space.

The decoder is effectively a model that learns to produce realistic data based on the representations in the latent space.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e783d6b4-cfd4-4ab3-842b-c607d0684304/Copy_of_Copy_of_Copy_of_Copy_of_noisy_latnet__720_x_400_px___2_.png)

It decodes latent variables by taking them and producing the parameters of a conditional distribution pθ(x|z). pθ(x|z) is the likelihood function modeled by the decoder, which defines how likely x is given z.

- The parameters of this function are given by a neural network.

During training, the reconstructed data x̂ is usually just the mean of pθ(x|z) for simplicity and stability when computing the loss.

#### Loss Function

VAEs are also unique by their loss functions. A VAE’s loss function has two parts instead of just one part.

The first part of the loss is the standard autoencoder reconstruction loss. This part ensures that the decoder learns to accurately reconstruct the input.

The second part of the loss is the Kullback-Leibler (KL) divergence. This is where the loss function gets a probabilistic twist.

The KL divergence measures how much the encoder’s learned distribution diverges from a standard prior distribution p(z). Essentially, it measures the difference between two probability distributions.

- The prior distribution represents our belief about how the latent variables z are distributed before observing any data x. The prior is typically a simple, well-known distribution—typically a standard normal distribution.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1262e4ef-8355-42a2-ad8a-a3b904c55899/image.png)

π1 and π2 are two distributions measured by the KL divergence

The model learns to minimize the difference between the encoder’s learned distribution and the chosen distribution, nudging the model to make a well-structured, smooth, and regular latent space. In other words, it is used to regularize q(z|x) to be close to p(z).

The KL divergence between the posterior q(z|x) and the prior p(z), for Gaussian distributions, can be computed as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/dea4bd8c-fe81-4959-89c5-7edea37d33a1/image.png)

Using the KL divergence and the reconstruction loss, we can build the VAE’s loss function. We call it the evidence lower bound (ELBO) loss, and its formula is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2fc9f840-d9d0-49e9-b1bd-78ea8aa19933/image.png)

Here, 𝔼 is the expectation operator, computing the average value of a log⁡(p(x|z)) over the distribution q(z|x), z∼q(z|x) indicates that z is sampled from the approximate posterior distribution q(z|x) (produced by the encoder), and KL represents the KL divergence function.

We update our parameters to minimize this ELBO loss, generating more accurate and consistent reconstructions every iteration.

#### Generation

Once the model is trained, we can generate new data by sampling from the prior distribution p(z) and then passing the sampled latent variable z through the decoder.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/acbf0fbc-d73b-4ee4-b41a-606ab8553a3d/image.png)

The prior is a standard normal distribution

The sampled z is fed into the trained decoder, which outputs the parameters of pθ(x|z). The generated data x̂ can now be obtained by sampling from pθ(x|z).

This process allows the VAE to generate new data points that are similar to the original data distribution, based on the learned latent space.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e486a224-95e5-4287-a3a1-2e748e65641e/image.png)

### Comparison

Variational autoencoders (VAEs) have continuous and smooth latent spaces, allowing interpolation and nice transitions of generated images. They have powerful generative capabilities to produce new but similar data.

Reconstruction loss is combined with KL divergence in its ELBO loss function, used to update the model’s parameters (mean and variance). The KL divergence pushes the latent space to follow a prior distribution, usually Gaussian.

Standard autoencoders have discrete, determinate latent spaces, which limits the model’s generative abilities. Its loss function is simply the reconstruction loss, helping the model reproduce the input accurately.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-visual-guide-to-variational-autoencoders-vaes):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-visual-guide-to-variational-autoencoders-vaes)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-visual-guide-to-variational-autoencoders-vaes)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team