# Optimizers in Deep Learning: A simple explanation #5 Jun 10, 2024 Training https://www.aibutsimple.com/p/optimizers-deep-learning-simple-explanation In deep learning, an optimizer is a crucial element that fine-tunes a neural network’s parameters during training. - An optimizer is simply an optimization algorithm (like gradient descent), and optimizers use different strategies to converge towards optimal parameter values for improved predictions Its primary role is to minimize the model’s error or loss function, boosting its accuracy and making it more useful in real world applications. But before talking more about optimizers, there are a few terms that you should be familiar with: - Epoch—Measure of time: One cycle through the entire training dataset - Batch—Subset of the training data - Batch size is the number of samples to be taken per update - Learning rate—Parameter that provides the model a scale of how much model weights should be updated. Also, if you don’t know what weights and biases are, or what a cost/loss function is, feel free to check out [this](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimizers-in-deep-learning-a-simple-explanation) previous issue for an explanation. In this issue, we won’t go too deep into the mathematical intuition of optimizers, so if you want to understand more about the inner workings of these optimizers, I suggest you check out some other resources. Let’s start off by talking about one of the most popular choices of optimizers, gradient descent. ### Gradient Descent --- This optimization algorithm uses calculus to modify the parameters to converge to the local minimum. Here are the basics: - It takes the gradient of the cost with respect to parameters (weights and biases) - It finds the new weights by subtracting the gradient multiplied by a learning rate ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/65d8e512-5f93-47a1-80b5-3e3a728f86d0/1_S3oLN40uWnGmP63lOFEDUg.gif) If you want a deeper dive into gradient descent, check out our previous [issue](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimizers-in-deep-learning-a-simple-explanation)about learning parameters (use the find tool and search “gradient descent”). But as good as gradient descent seems, there can be some issues when using this optimizer. - For instance, it can get very computationally expensive to compute the gradients if the data is very large Also, gradient descent works well for convex functions, but when it comes to non-convex and more complicated functions, it may not behave the way we want. For instance, it can get stuck at a local minimum or at saddle points, and end up not finding its way down to the absolute minimum. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fa3902e6-107a-4db3-b5cc-96fb117f8e5d/image.png) Non-convex functions pose a large problem to gradient based optimization algorithms like gradient descent, and new optimizers tackle this problem in various ways. One simple way is to add some randomness and noise to the convergence, like in Stochastic Gradient Descent: ### Stochastic Gradient Descent (SGD) --- To tackle the challenges large datasets and non-convex functions pose to gradient descent, we can use stochastic gradient descent. - The word stochastic represents the element of randomness used in the algorithm. In stochastic gradient descent, instead of processing the entire dataset during each iteration, we randomly select a sample (random shuffle) of data and process it to update our parameters. This means that only one sample from the dataset is considered at a time, allowing for more efficient and computationally feasible optimization. However, since we are not using the whole dataset but batches of it for each iteration, the path taken by the algorithm is full of noise as compared to the gradient descent algorithm. - This noise can hinder its ability to converge quickly, but it can be useful in some cases - For instance, this noise can be beneficial to escape from some local minima and valleys (a problem in gradient descent) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6887a84c-39fe-44af-823c-fcc9054f8dfe/image.png) Because of this noise, SGD uses a higher number of iterations to reach the local minima. But if the data is huge, stochastic gradient descent should be preferred over batch gradient descent (also known as gradient descent), since it may be able to converge faster even after increasing the number of iterations. ### Momentum --- Stochastic gradient descent takes a much more noisy path, so it requires a significantly greater number of iterations to reach the optimal minimum, so computation time can be quite slow. To overcome this problem, we can use SGD with a momentum algorithm. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e8c58ca3-5603-4311-957d-f22eb9418c8c/image.png) This momentum helps the model converge to the minimum of the loss function quicker: - It does this by adding a portion of the previous update to the current update, which makes the steps larger in the direction of the minimum (sort of pushes it down) But remember that while increasing the momentum, the possibility of passing the optimal minimum also increases (which could give low accuracy) - So when using this optimizer, the learning rate should be decreased with a high momentum term, or else it may overshoot the minimum. ### Mini-Batch --- In this variant of gradient descent, it will split the data into smaller batches (mini-batches) and update the model parameters after processing each mini-batch. Since we are using a batch of data instead of taking the whole dataset, we need less iterations before an update. Mini-Batch is typically faster than Batch Gradient Descent because it updates the model parameters more frequently. It can also converge faster than SGD because the gradient estimates are less noisy. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d4179944-759f-4243-adbc-c4611c12f89d/image.png) In practice, Mini-Batch Gradient Descent is generally preferred since it provides a good balance between the stability of Batch Gradient Descent and the speed of SGD. - However, it needs a hyperparameter called “batch-size”, which needs to be tuned to achieve a lower loss. ### RMS Prop (Root Mean Square Propagation) --- Root Mean Square Propagation is one of the most popular optimizers among deep learning enthusiasts. It actually hasn’t been published but is still very well-known in the community. For RMSprop, it mainly focuses on making the optimization process faster by decreasing the number of function evaluations to reach the local minimum. The algorithm keeps track of the moving average of squared gradients for every weight and divides the gradient by the square root of the mean squared. - The step size for each parameter is adjusted based on the moving average of the squared gradients. In simpler terms, if there exists a parameter due to which the cost function oscillates a lot, we want to penalize the update of this parameter (so it can rely on other features). This prevents the algorithm from adapting too quickly to changes in a certain parameter compared to other parameters. The algorithm converges quickly and requires less tuning than gradient descent algorithms and their variants. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e6011274-e1c5-4e68-8ef9-545709ed9796/1_Y2KPVGrVX9MQkeI8Yjy59Q.gif) ### Adam Optimizer --- Adam (short for Adaptive Moment Estimation) is an optimization algorithm that combines the advantages of two other popular optimization algorithms: AdaGrad and RMSprop. It can adaptively adjust the learning rate for each network weight individually. Unlike SGD, which maintains a single learning rate throughout training, Adam optimizer dynamically computes individual learning rates based on past gradients. - It does this by using moving averages of both the gradients (first moment) and the squared gradients (second moment) to adapt the learning rate. Adam has become a benchmark for deep learning papers and is recommended as a default optimization algorithm due to its popularity and performance. Moreover, the algorithm is straightforward to implement, has a faster running time, has low memory requirements, and requires less tuning than any other optimization algorithm. Of course, there are many other optimizers out there, and I encourage you to learn more about them. But for now, these are some of the most popular optimizers that are being used! --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=optimizers-in-deep-learning-a-simple-explanation) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team