KL Divergence, Simply Explained

KL Divergence can be simply described as a method used to compare the difference between two probability distributions.

In modern machine learning, this “difference” determines the quality of our predictions.

By comparing the probability distributions of the learned and known true outputs, we determine how “close” or accurate the learned distribution is to the true one.

For instance, two different probability distributions are shown below—the predictions are somewhat off from the true labels.

But how exactly is this difference between distributions measured? In this issue, we will uncover and understand this clearly as we dive into KL divergence and its usage in machine learning.