# Deep Learning: Overfitting, Underfitting, and Vanishing Gradient #3 May 27, 2024 Training https://www.aibutsimple.com/p/deep-learning-overfitting-underfitting-vanishing-gradient Last week, we discussed how a neural network uses backpropagation and gradient descent to adjust its weights. - (If you haven’t read it, you can find it [here](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-overfitting-underfitting-and-vanishing-gradient)) But when a neural network is learning, if you either let it run for a very long time, or the training data is very skewed or small, you may encounter something called overfitting. - Simply explained, overfitting is when the model learns the specific noise of the dataset too well, causing it to perform badly on new data. Things like overfitting and many other issues can occur while training, but the problem doesn’t stop there. So today, we’re going to discuss some common issues that neural networks encounter during training, and how to fix them. ### Overfitting and Underfitting In training, it is common practice to split your dataset into three parts: a training set, a validation set, and a testing set. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6e93c6b3-a582-45a2-bed9-5612a116bca8/Add_a_heading__9_.png) However, when the model trains on the training data set for too long or when the model is too complex, it can start to learn the “noise”, or irrelevant information, within the dataset. When the model memorizes the noise and fits too closely to the training set, it starts overfitting, and it is unable to generalize well to new data. If a model cannot generalize well to new data, then it will not be able to perform the classification or prediction tasks with an accuracy it was intended for. In summary, overfitting is when a model has high accuracy (low error) on the training data set, while having high error on the test set/new data. So how do we know if a model is overfitting? When you train the model and note down its performance over its training cycles, if it performs extremely well on the training data but poorly on the validation or test data, it might be overfitting. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bda587e2-bc1d-44aa-bd9f-93d55e6a1f7c/image.png) Performance on test data sloping down Another indicator of overfitting is when the training accuracy increases over time while the validation accuracy makes a plateau or decreases. - It’s getting better at the specific details about the training set, so its performance on new data will stay the same or decrease But there’s also a visual way to see if the model is overfitting. Consider the cases below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d6eb120b-b4a1-4674-9886-996b08d33a21/image.png) In classification, if outliers are taken too much into account, it will fail to classify new data to a high accuracy. Likewise, in regression, if there are jagged line segments connecting each point, it will not perform as well as a smooth curve for new data. Now what about underfitting? A neural network underfits when it fails to learn the underlying patterns in the training data. This typically occurs when the model is too simple to represent the data. - For instance, the network might not have enough layers or neurons to capture the complexities of the data. Another reason a model might underfit is that it might not have been trained for enough epochs (training cycles) to learn the underlying patterns. There may also be too little training data, or the data might not be representative of the problem space. There are other reasons for this, such as feature selection and unfit regularization, but we'll stick to these simple ideas for now. So how can we tell if a model is underfitting? Again, when noting down the model’s performance over its cycles (epochs): - If there is a high training error that doesn’t lower over many epochs, it may be underfitting. Another way to tell is that if both training and validation errors are high and close to each other, it tells you that the model is not learning the underlying patterns, hence overfitting. Here’s a visual display: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5cbfe13a-01d5-433b-b696-c86ca33a7dee/image.png) Training a neural network is an optimization problem So how can we fix underfitting? One way to mitigate underfitting is to add more layers or neurons to the network, if the model is too simple. If the model hasn’t trained for long enough, we can also increase the training time, as in train the model for more epochs. It might even be a dataset issue: - The model might benefit from gathering more data to learn the features from. Other and more advanced methods include feature selection and reducing the impact of regularization. ### Vanishing Gradient Another problem that can occur during training is called a vanishing gradient. Vanishing gradients happen while training deep neural networks using gradient-based optimization methods (one of which, is gradient descent). - It occurs due to the nature of the backpropagation algorithm that is used to train the neural network. One of the main reasons for which vanishing gradients occur is the type of activation function chosen. - Activation functions such as sigmoid and tanh are known to cause this problem, since their gradients are very small when the input is near the tails of the function. Consider the graph of the sigmoid function and its derivative: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/173f470c-5f2e-4646-821c-0e847bb3f254/image.png) For very large inputs, observe that the derivative of the sigmoid function is a very small value. If the neural network has many hidden layers, the gradients in the earlier layers will become very low as we multiply the derivatives of each layer. As a result, learning patterns slow as not much data is being passed to the other layers. So, how can we fix this? Now, let’s look at using a ReLU activation function. The derivative of the ReLU function is either 0 or 1. Even if we multiply the derivatives for many layers, there will be much less diminishing of the gradient, as the derivative of ReLU and its similar family of activations will be 1 whenever the input is positive. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a7e818f1-1900-4438-8394-7989bc97e3e2/image.png) Leaky ReLU is a variant of ReLU to improve performance in certain cases. However, a ReLU is not foolproof. If the inputs are constantly negative, the gradients will diminish slowly. - That’s why variants of ReLU have been created, such as Leaky ReLU, ELU, PReLU, GELU, etc. And that just about sums up some common problems when training neural networks. Hope this added to your intuition about deep learning! --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-overfitting-underfitting-and-vanishing-gradient) Advertising, thoughts, suggestions, feedback? Please send to [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). --- That’s it for the third issue of AI, but simple. See you next week! —AI, but simple team