Deep Learning: Overfitting, Underfitting, and Vanishing Gradient

Last week, we discussed how a neural network uses backpropagation and gradient descent to adjust its weights.

• (If you haven’t read it, you can find it here)

But when a neural network is learning, if you either let it run for a very long time, or the training data is very skewed or small, you may encounter something called overfitting.

• Simply explained, overfitting is when the model learns the specific noise of the dataset too well, causing it to perform badly on new data.

Things like overfitting and many other issues can occur while training, but the problem doesn’t stop there.

So today, we’re going to discuss some common issues that neural networks encounter during training, and how to fix them.

Overfitting and Underfitting

In training, it is common practice to split your dataset into three parts: a training set, a validation set, and a testing set.

However, when the model trains on the training data set for too long or when the model is too complex, it can start to learn the “noise”, or irrelevant information, within the dataset.