# Regularization in Deep Learning

#7 Jun 24, 2024 Training

https://www.aibutsimple.com/p/regularization-deep-learning

One of the most common problems encountered by deep learning enthusiasts and professionals is the problem of overfitting.

Overfitting can lead to poor performance on new data, especially in the presence of outliers or noise in the training set.

This has become even more significant in the last 5 years, as neural network architectures have become extremely complicated (which make them prone to overfitting).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f98a7f1f-d4c3-40fc-ae7a-0245f3661626/image.png)

As the complexity increases, the model tends to overfit. The overfitting/underfitting problem is also called the bias-variance tradeoff.

There are many ways to prevent overfitting in the world of deep learning, and one of the most popular ways to do so is called regularization.

Regularization is a technique used in machine learning and deep learning to prevent overfitting and improve the generalization of a given model.

- When a model fits the noise of the dataset too well, it will have performance issues on new data (only performing well for the train dataset)
- If you want to learn more about overfitting, read this previous [issue](https://www.aibutsimple.com/p/deep-learning-overfitting-underfitting-vanishing-gradient?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=regularization-in-deep-learning)

Regularization involves adding a “penalty term” to the loss function when the model is training.

- This penalty prevents the model from becoming too complex or having large weights, which helps control the amount of noise a model will fit in a dataset.

Some common regularization methods used frequently in deep learning are L1 and L2 regularization, dropout, early stopping, and more.

- Through regularization, models become more robust and better at making accurate predictions on unseen data.

Let's dive into some of the most common regularization methods used in deep learning, starting with L1 and L2 regularization:

### L1 & L2 Regularization

---

L1 and L2 are the most common types of regularization in deep learning. These methods update the cost function by adding another term to reduce the probability of overfitting.

In deep learning, regularization sort of “penalizes” the weight matrices of the nodes.

- There is a certain coefficient, called a regularization coefficient, which is used in both L1 and L2 (the coefficient is a parameter that needs to be tuned)
- If the value of the coefficient is too high, the model will start underfitting, as the model will become more linear, but when it is too low, the regularizing effect will not be as strong.

L1 and L2 regularization take the cost function and add an additional regularizing term.

- Specifically, L1 regularization involves adding the absolute values of the weights to the cost function, while L2 regularization involves adding the squared values of the weights to the cost function.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/da243e6a-ebca-4ffc-97cd-058ada7d14c6/image.png)

L1 Regularization involves the absolute value, while L2 involves the square. Regularization coefficient is shown as Lambda.

This regularization term is the weight matrix multiplied by the regularization coefficient (lambda).

- Since the coefficient is typically less than 1, it diminishes the values of the weights in the cost function.
- That’s useful since models with smaller weights often generalize better and reduce overfitting.

Regularization not only affects the cost function but the weights as well, and updating the weights will diminish them.

During the gradient descent step, weights are updated as follows:

- Without Regularization:
  ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/52290689-fa30-4829-9ee6-2651f3354e6a/image.png)
  where η is the learning rate, and Θ is the weight matrix
- With Regularization:
  - L1 Regularization:
    ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/aa5cd00a-9b9e-4cb2-8719-28cd4e190e87/image.png)
    - So it subtracts more since there is an additional regularization term
  - L2 Regularization:
    ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d417817d-4ecc-4dcc-887c-2f61159a4c30/image.png)

### Dropout

---

This is a seemingly simple yet clever way to regularize a model. It produces very good results and is very frequently used in the field of deep learning.

To understand dropout, let’s say our neural network structure is similar to the one shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a1d5839a-50aa-415e-992e-4fcc88a28cad/Copy_of_Add_a_heading__1___1_.png)

At every iteration, dropout randomly selects some nodes and removes them along with all of their incoming and outgoing connections (setting the neuron’s outputs to 0):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/65a2a2dd-8578-4d5d-8c3e-b8c801a29352/Copy_of_Add_a_heading__2_-1.png)

In each iteration, the model has a different set of nodes, which results in a different set of outputs. It’s similar to ensemble techniques in machine learning.

- This probability of a node being dropped is the hyperparameter of the dropout function (which can be tuned).

### Data Augmentation

---

The simplest way to reduce overfitting is to increase the size of the training data.

But how can we do this without spending more time and resources gathering new data?

- We can use old data, tweak certain things, and create variations of some past data points.

This way, we’ll not only increase the size of the data, we’ll also add to the diversity of the dataset.

For instance, let’s say we are dealing with images.

In this case, there are a few ways of varying the data– rotating the image, flipping, scaling, shifting, etc:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9a0f5ec5-bf31-4051-b5a4-e059f691bdcb/image.png)

This way, we can make multiple new data points from one single point.

This method is known as data augmentation. This usually provides better accuracy and lower loss on a general case.

### Early Stopping

---

Early stopping is one of the most simple and intuitive regularization methods.

We monitor the model's performance on a validation dataset during training, and we stop the training process when the performance on the validation set starts to deteriorate.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/91776e3e-ec4e-4bd3-856c-239cfe7ed4d8/image.png)

Early stopping helps prevent the validation loss from ramping back up.

In many libraries, early stopping will have a patience parameter, which denotes the number of epochs that see no improvement to the validation error before stopping.

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=regularization-in-deep-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team