Generative Adversarial Networks, Simplified

Imagine teaching two clever students who have very different talents to improve their skills by challenging each other while you let them figure things out on their own. That is the fundamental idea behind Generative Adversarial Networks (GANs), a generative model that has revolutionized how we think about generating data.

Most of the time, standard neural networks are used for making predictions. In these models, we feed in data, transform the data, and produce a predictive output that picks a class or value.

This kind of learning is called discriminative learning, as in, we’d like to be able to discriminate between a spam email and a non-spam email. Classifiers and regressors are both examples of discriminative learning.

Neural networks trained through backpropagation have transformed our ability to handle large, complex datasets, making discriminative learning extremely important in modern deep learning applications.

However, while discriminative learning has made huge strides, it has its limitations. Until about 10 years ago, generative tasks (those that involve creating new data, like generating “real-looking” images) remained an underdeveloped area, yet still needed for the world of deep learning.

Mathematically speaking, what’s the difference between generative learning and discriminative learning?

Let's say you have input data x, and you want to classify the data into labels y.

A generative model learns the joint probability distribution p(x,y), while a discriminative model learns the conditional probability distribution p(y|x).

• The conditional probability p(y|x) is the probability of y given x.

In simple terms, a generative algorithm models how the data was generated in order to label an input. It asks the question: given how the data is generated, which category is most likely?

A discriminative algorithm does not care about how the data was generated; it simply categorizes a given input.

But the success of deep neural networks in discriminative learning led to new possibilities. Over the past few years, researchers have started applying discriminative models in ways we wouldn't traditionally consider “supervised learning”.

In 2014, a breakthrough paper introduced Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), a clever new way to leverage the power of discriminative models to create good generative models.

The core idea behind GANs is simple yet very powerful: a data generator is considered successful if it can produce data that is not distinguishable from real data.