# KL Divergence, Simply Explained

#63 Aug 11, 2025 Theory

https://www.aibutsimple.com/p/kl-divergence-simply-explained

KL Divergence can be simply described as a method used to compare the difference between two probability distributions.

In modern machine learning, this “difference” determines the quality of our predictions.

By comparing the probability distributions of the learned and known true outputs, we determine how “close” or accurate the learned distribution is to the true one.

For instance, two different probability distributions are shown below—the predictions are somewhat off from the true labels.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c89bdeab-e92c-49cf-9e4a-a52d0ea6be55/Copy_of_True_Labels.png)

But how exactly is this difference between distributions measured? In this issue, we will uncover and understand this clearly as we dive into KL divergence and its usage in machine learning.

### KL Divergence in Machine Learning

Modern machine learning can be thought of as basically learning probability distributions from data.

Whether we are training neural networks for object detection in images or building a language model for generating text, we are essentially trying to approximate unknown probability distributions based on input data.

Before we try to understand what exactly KL Divergence is and how it works, let’s first take a quick look at where exactly in ML you may encounter this method so that its importance can be evident right away.

For example, in supervised learning, cross-entropy loss functions are applications of KL Divergence.

While training a classification neural network, we try to minimize the KL divergence (the difference) between the predicted probability distribution and the distribution of the true labels.

On the other hand, in unsupervised learning, KL divergence helps in the learning of latent representations for [Variational Autoencoders (VAEs)](https://www.aibutsimple.com/p/a-visual-guide-to-variational-autoencoders-vaes?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8711d7ae-a63c-46df-b4aa-4736e76ed9c9/Copy_of_P_ZX___1_.png)

KL Divergence in VAEs

In reinforcement learning, KL divergence appears in policy gradient methods, where it is used to set up constraints ensuring that policy updates don’t deviate too far from the current policy.

---

### A Private Circle for High-Net-Worth Peers

[Long Angle](https://www.longangle.com/join-community-bh/?utm_campaign=XIAEVJHTGJ&utm_source=beehiiv&_bhiiv=opp_123fec92-67eb-4799-8c10-06aa6bb3b327_b43db979) is a private, vetted community for high-net-worth entrepreneurs, executives, and professionals across multiple industries. No membership fees.

Connect with primarily self-made, 30-55-year-olds ($5M-$100M net worth) in confidential discussions, peer advisory groups, and live meetups.

Access curated alternative investments like private equity and private credit. With $100M+ invested annually, leverage collective expertise and scale to capture unique opportunities.

[Apply to Join](https://www.longangle.com/join-community-bh/?utm_campaign=XIAEVJHTGJ&utm_source=beehiiv&_bhiiv=opp_123fec92-67eb-4799-8c10-06aa6bb3b327_b43db979)

---

### Information Theory Foundation

To understand how KL Divergence was developed, we need to understand its origins in [Information Theory](https://en.wikipedia.org/wiki/Information_theory?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained). For this purpose we will first go through the concepts of Entropy and Cross-Entropy.

Information Theory and its concepts were introduced by [Claude Shannon](https://en.wikipedia.org/wiki/Claude_Shannon?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained), an American mathematician and electrical engineer, in his 1948 paper “[A Mathematical Theory of Communication](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained).”

He did this to find a solution for transmitting messages over noisy channels.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bb542537-1728-4cc9-b98b-1b6376332b59/Information_Theory.png)

For statistics and machine learning, in simple terms, Information Theory gives us the tools to understand how well our learned distributions match the true underlying distributions.

It inherently helps us answer how well our model captures the patterns in the training data.

### Entropy

Before we understand how to compare probability distributions, we need to understand how to measure the information content of a single probability distribution.

Shannon entropy gives us the average amount of information contained in a probability distribution, measured in [Nats](https://en.wikipedia.org/wiki/Nat_(unit)?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained) or [Bits](https://en.wikipedia.org/wiki/Shannon_(unit)?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained) (1 nat = 0.69 bits).

For a discrete probability distribution P with outcomes x1, x2, …, xn with probabilities p1, p2, ..., pn, the entropy (H) is given by:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5f15693a-002f-437b-83e3-4556efddb605/image.png)

Entropy helps us measure the average uncertainty in a discrete/continuous random variable. Uncertainty in the random variable increases entropy, and the more predictable an outcome the less entropy it contains.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a16272da-7895-4037-91d7-fd3029583711/Entropy_Visualization.png)

For example, flipping a fair coin (a fair coin has all equal probabilities of outcomes) has maximum entropy, as it is quite uncertain what the outcome will be between a head or tails.

But a biased coin that always lands on tails has zero entropy because there is no uncertainty—we know that it will land on tails.

This entropy concept is also used in [decision trees](https://www.aibutsimple.com/p/supervised-ml-decision-trees?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained) but measures the uncertainty of a random variable rather than disorder.

### Cross-Entropy

Cross-entropy extends the concept of entropy to measure the average number of bits required to identify an event (an outcome with a probability) from one probability distribution by using another probability distribution.

If we use a distribution Q that is poorly approximated to our true distribution P, the cross-entropy tells us how poor or inefficient our approximation Q is. This is why it is used as a common loss function in machine learning.

Mathematically, cross-entropy is defined as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7d4dd57c-403c-4286-ac96-4ec41867495b/image.png)

Here, pi are the true probabilities, and qi are the approximated probabilities.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained)

---

### KL Divergence Definition

The Kullback-Leibler divergence, named after [Solomon Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained) and [Richard Leibler](https://en.wikipedia.org/wiki/Richard_Leibler?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained), is a natural measure for the difference/distance between probability distributions.

It is motivated by looking at how likely the second distribution would be able to generate samples from the first distribution.

#### Why is this helpful?

Modern machine learning and deep learning are a lot about modeling true underlying distributions. KL divergence becomes a very important tool used in loss functions to analyze the performance of predicted distributions versus the true ones.

In fact, the very common cross-entropy loss is very similar to KL divergence, where minimizing the cross-entropy is equal to minimizing the KL divergence between distributions when the true distribution is fixed.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7ba97b90-d845-47c5-afc8-20892e06b546/KL_Divergence.png)

#### What's the difference between cross-entropy and KL divergence then?

It should be noted that although these two concepts seem similar, they are used for different purposes in AI.

Cross-entropy is used in ML in loss functions for classification tasks to minimize the error between the probability distribution of the predictions and the probability distribution of the true labels.

In contrast, KL divergence is used in Unsupervised Learning models such as [Autoencoders](https://www.aibutsimple.com/p/deep-unsupervised-learning-autoencoders?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained), where the goal is to learn the patterns in the data by minimizing the difference between the true and learned distributions. This helps an autoencoder uncover the structure of the data.

### Calculating KL Divergence

For discrete distributions P and Q, KL divergence is defined as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ea4daee-7f74-42c8-93aa-7b7551749c53/image.png)

The double bar notation is specific to KL divergence and denotes a divergence instead of a conditional probability.

The above, using the rules of logarithms, can be written as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6dd75412-1a77-498f-8c8d-6d9017c186a0/image.png)

This gives us the KL divergence of distribution Q from P.

For continuous distributions, the sum is replaced with an integral:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f456129d-9ad1-44cb-9b81-51ce60ba484a/image.png)

It is important to remember that KL divergence is not symmetric—DKL(P||Q) is not equal to DKL(Q||P).

This is because it measures how much information is lost when Q is used to approximate P, which is different from the information lost when P is used to approximate Q.

### Simple Example: Comparing Weather Models

Let’s go through an example of KL divergence in practice. Consider two weather prediction models showing probabilities of daily rainfall.

Model P (true model) predicts:

- No rain: 0.7
- Light rain: 0.2
- Heavy rain: 0.1

Model Q (approximation) predicts:

- No rain: 0.6
- Light rain: 0.3
- Heavy rain: 0.1

Now, let’s calculate DKL(P||Q):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/151c951f-7f6d-4cdf-9b96-f1425a47939f/image.png)

This shows that using the model Q instead of the true model P requires 0.027 additional nats (additional information) per prediction on average.

### Conclusion

From its roots in Shannon’s information theory to its modern applications in deep learning and artificial intelligence, KL divergence has connections to entropy, cross-entropy, and maximum likelihood estimation, revealing deep relationships between different concepts in statistics and machine learning.

As machine learning and AI continue to evolve, KL divergence provides a mathematical foundation for measuring and minimizing the difference between what we know and what we want to learn. KL divergence reflects the important role of probability and information theory in AI, helping us build smarter and more efficient methodologies!

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=kl-divergence-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team