Generative Adversarial Networks, Simplified

Imagine teaching two clever students who have very different talents to improve their skills by challenging each other while you let them figure things out on their own. That is the fundamental idea behind Generative Adversarial Networks (GANs), a generative model that has revolutionized how we think about generating data.

Most of the time, standard neural networks are used for making predictions. In these models, we feed in data, transform the data, and produce a predictive output that picks a class or value.

This kind of learning is called discriminative learning, as in, we’d like to be able to discriminate between a spam email and a non-spam email. Classifiers and regressors are both examples of discriminative learning.

Neural networks trained through backpropagation have transformed our ability to handle large, complex datasets, making discriminative learning extremely important in modern deep learning applications.

However, while discriminative learning has made huge strides, it has its limitations. Until about 10 years ago, generative tasks (those that involve creating new data, like generating “real-looking” images) remained an underdeveloped area, yet still needed for the world of deep learning.

But the success of deep neural networks in discriminative learning led to new possibilities. Over the past few years, researchers have started applying discriminative models in ways we wouldn't traditionally consider “supervised learning”.

In 2014, a breakthrough paper introduced Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), a clever new way to leverage the power of discriminative models to create good generative models.

The core idea behind GANs is simple yet very powerful: a data generator is considered successful if it can produce data that is not distinguishable from real data.

This concept reflects a statistical technique called a two-sample test, which asks whether two datasets come from the same distribution.

However, the main difference between most statistics papers and GANs is that GANs use this idea in a constructive way.

In other words, rather than just training a model (called a discriminator) to say, “Hey, these two datasets do not look like they came from the same distribution,” they use the two-sample test to give training signals to a generative model.

This allows us to improve the generator until it generates something that resembles the real data. At the very least, it needs to fool the discriminator.