# Optimization Algorithms in Deep Learning: Mathematically Explained

#24 Nov 4, 2024 Training

https://www.aibutsimple.com/p/optimization-algorithms-in-deep-learning-mathematically-explained

An optimization algorithm in deep learning is a method used to adjust the parameters (weights and biases) of a neural network to improve its predictions.

These algorithms are based on loss functions, functions used to measure how good or bad a given prediction is.

- It calculates the difference between the network's prediction and the actual result.
- If the output is 5, and the real value is 3, the loss is going to be greater than if the output was 4.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5ad2d8ab-bfc9-4124-903f-0a2e3a650320/image.png)

Some examples of loss functions

The goal of optimization is to reduce this error (or loss) as much as possible. Reducing this loss happens through shifting the numerical values of the parameters, and it makes the model’s predictions more accurate.

The optimization process stops when the algorithm reaches a point where the error can’t get any lower (or only gets lower very slowly). At this point, we say the model has converged. Ideally, this means the network is now as accurate as it can be on the given data.

A quick side note: you may have read certain publications or tutorials that discuss the cost function, the objective function, and the loss function as 3 different things.

In our issues, we use these terms interchangeably, but for context:

- The loss function measures the error for a single training example.
- The cost function is the average of the loss function over the entire dataset. It gives an overall measure of the model’s performance.
- The objective function is the function that the optimization algorithm is trying to maximize/minimize, which can include the cost function and additional terms from [regularization](https://www.geeksforgeeks.org/regularization-in-machine-learning/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/579d1684-683e-4b91-b216-84f1f44b6d0d/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__2_.png)

Technically speaking, an optimization algorithm optimizes the objective function, but it’s easier to think about the loss function (since it’s the distance between the actual and predicted results, which is the thing we want to minimize).

For a given dataset, an ideal optimization algorithm will navigate iteratively through a highly complex, multidimensional parameter space, adjusting each parameter to reach the lowest possible error.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/39bfa79a-6061-4d46-8147-17b405f01828/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__1_.png)

Due to the mathematical nature of this issue, before continuing, it is highly recommended to have some background knowledge on multivariable calculus and an overall mathematical understanding of neural networks (as a layered structure that uses mathematical transformations to compute a relevant output).

Here are some issues to help with the intuition of neural networks:

- [MLPs, Mathematically Explained (Beginner to Intermediate)](https://www.aibutsimple.com/p/multilayer-perceptrons-math-behind-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)
- [Neural Networks, Explained (Beginner)](https://www.aibutsimple.com/p/neural-networks-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)
- [Neural Network Learning (Beginner)](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)

Also, if you want to learn more about the mathematics of gradient calculations (including backpropagation), feel free to check out these issues:

- [Recurrent Neural Network Backwards Pass (Intermediate)](https://www.aibutsimple.com/p/mathematics-of-recurrent-neural-networks-backpropagation?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)
- [MLP Backwards Pass (Intermediate)](https://www.aibutsimple.com/p/mlp-learning-process-math-behind-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)

### Optimization Terms

---

Here are some terms you need to know:

- Epoch—Measure of time: One cycle through the entire training dataset.
- Batch/Minibatch—Subset of the training data.
  - Batch size is the number of samples to be taken per update.
- Learning rate—Parameter that provides the model a scale of how much model weights should be updated.

### Gradient Descent

---

Gradient Descent (GD) is the foundational technique that most optimization algorithms are based on. In its basic form, gradient descent iteratively updates the parameters, trying to converge to some minimum (local or global).

The update rule for gradient descent is as follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7fe238f1-60b2-4ce0-ab1d-ff4bc58e1405/image.png)

Where W are the weights, t references the iteration in training, α represents the learning rate, and ∂𝐿/∂𝑊 is the partial derivative of the loss with respect to weights.

We update 𝑊 by moving it in the opposite direction of the gradient (subtract the gradient) to minimize the loss. By subtracting the gradient, it converges to a minimum (see the below image).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/266db07b-232e-444b-81c8-26638dd3e00c/image.png)

The learning rate (α) is a set value (a hyperparameter), and the old weight matrix is already known. But how do we obtain the gradient of the loss with respect to weights?

In a mathematical lens, the loss function (denoted L) measures the difference between the model’s predictions and the true values.

For example, in a neural network, the loss function might look like this for a single training example:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2e25349e-84b0-4f16-a48a-6ec74f281110/image.png)

The loss function above calculates the square difference between the predictions and the actual values.

Something to note is that the loss function does not directly contain the weights in its function definition, but it’s still useful, and we’ll discuss it later.

Let’s talk about the output or prediction, denoted as ŷ.

For a simple network with one layer, the output could be calculated like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0640848c-0d8d-4af7-8bc9-6b03d1efce7c/image.png)

Remember, neural networks obtain their prediction by applying mathematical transformations to input data. In this case, the input data is multiplied by a weight matrix, which is then added to some bias vector, and all of this is simply a linear transformation.

To find out how changes in the weights W affect the loss L, we use the chain rule in calculus. The chain rule allows us to compute the derivative of L with respect to W by breaking it down into intermediate steps.

Since the loss (L) is a function of the prediction (ŷ), and ŷ​ is a function of the weights (W), by the chain rule, we can write:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/246b3973-9825-4db7-ab66-8d7a2730a200/image.png)

By multiplying these two partial derivatives, we get the overall gradient ∂L/∂W​, which shows how the loss depends on the weights indirectly through the model’s predictions.

We can apply the same method to obtain the loss of biases and other parameters.

Here is an example of gradient descent in action:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f378a45b-a15a-44db-84c2-e8f66d295788/gradient_descent_parameter_a.gif)

As you can see above, the parameters are tweaked iteratively (on the right plot) by subtracting the negative gradient multiplied by some learning rate.

On the left, we observe that every time the parameter a is tweaked, it gets a little closer to the minimum.

This descending motion through parameter changes is the foundation that led to many other optimization methods to be created.

Let’s talk about why gradient descent ended up getting replaced by better algorithms.

Although gradient descent can work well in certain cases, it’s usually used only since it’s easy to understand and implement.

Gradient descent has certain problems that bottleneck its performance.

For instance, sometimes within the parameter space, there will be saddle points or plateaus. Saddle points and plateaus are points or areas where the gradient is zero (or close to zero) but is not a minimum.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95ed0e11-c91e-40af-a5e9-152104f522d9/image.png)

- When using gradient descent, the algorithm can get stuck in these flat regions since the gradient is 0 or close to 0; this leads to the parameters being unchanged as there is almost nothing being subtracted.

To solve this issue, we can use momentum, which is introduced in newer optimization algorithms.

Gradient descent is also vulnerable to the vanishing/exploding gradient problem. However, this problem is solved by regularization or using different activation functions.

Finally, the biggest issue with gradient descent is that it is inefficient and it doesn’t perform that well.

Standard gradient descent updates parameters only once after running through the entire dataset (once per epoch) since it uses the whole dataset to calculate the gradient of the loss with respect to certain parameters.

Researchers were looking for ways to speed up the convergence process, and one of the ideas was to update the weights and biases more frequently.

### Stochastic Gradient Descent

---

Instead of updating the parameters after finishing the whole dataset, we can update the parameters at every single training example (many times per epoch).

This is exactly what Stochastic Gradient Descent (SGD) does, and it can help when dealing with large datasets or unusual parameter spaces.

- Updating the parameters frequently can also lead to some issues, like noise and being sensitive to outliers, which creates a jagged path downwards to the minimum.
- This noise can be beneficial to escape from some local minima and valleys in the parameter space.

The update rule for SGD is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/53d4e590-9e5c-481d-930a-883a663d2795/image.png)

Where (x(i),y(i)) is a randomly chosen data point. W are the weights, t references the iteration in training, α represents the learning rate, and ∂𝐿(x(i),y(i))/∂𝑊 is the partial derivative of the loss with respect to weights at the given data point.

Note that SGD looks almost identical to standard gradient descent (we call it batch gradient descent). The only difference is using the derivative of loss with respect to weights at one randomly chosen point.

Let’s see how GD and SGD are different mathematically.

Because the loss function (technically the cost function) is calculated over the sum of all data points, the partial derivative calculation will be affected:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/979417ed-680d-434f-9d38-85097bd79e24/image.png)

The loss function used here is Mean Squared Error (MSE)

Don’t worry about the derivation of these derivatives; they are found using the chain rule explained previously.

Above, you see that gradient descent (GD) will calculate the derivative using the sum over all training samples, where N is the number of samples, finally dividing it by N.

On the other hand, SGD simplifies the derivative since N = 1 (it uses one sample at a time), allowing the loss to be calculated earlier and the parameters to be updated quicker.

### Mini-Batch Gradient Descent

---

Instead of updating the parameters every time after computing the output for one training sample, we might choose to update them after computing a small number of training examples, called a mini-batch (usually 32, 64, 128, etc.).

The update rule would be like something like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/91e30fed-7cd3-47c1-84a5-9d4399137ac8/image.png)

Where m is the mini-batch size. The mini-batch size is also called the batch size, and it is a hyperparameter that the user sets and tunes. In the same fashion as stochastic gradient descent, W are the weights, t references the iteration in training, α represents the learning rate, and ∂𝐿(x(i),y(i))/∂𝑊 is the partial derivative of the loss with respect to weights at the given data point.

The only difference between this algorithm and stochastic gradient descent is that it sums all the losses in the batch and divides it by the batch size.

- Since we’re using mini-batches, m will be smaller than the whole dataset, so we can still take advantage of quicker updates.

Here’s a side-by-side comparison of SGD, Mini-batch GD, and GD:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f27626c8-b876-442f-a623-05ac21aaea85/image.png)

There are many other modern optimization algorithms, but we won’t cover them in this issue.

Hopefully you learned a thing or two about optimization algorithms and their mathematical foundations!

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimization-algorithms-in-deep-learning-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team