Optimization Algorithms in Deep Learning: Mathematically Explained

An optimization algorithm in deep learning is a method used to adjust the parameters (weights and biases) of a neural network to improve its predictions.

These algorithms are based on loss functions, functions used to measure how good or bad a given prediction is.

• It calculates the difference between the network's prediction and the actual result.

• If the output is 5, and the real value is 3, the loss is going to be greater than if the output was 4.

The goal of optimization is to reduce this error (or loss) as much as possible. Reducing this loss happens through shifting the numerical values of the parameters, and it makes the model’s predictions more accurate.

The optimization process stops when the algorithm reaches a point where the error can’t get any lower (or only gets lower very slowly). At this point, we say the model has converged. Ideally, this means the network is now as accurate as it can be on the given data.

A quick side note: you may have read certain publications or tutorials that discuss the cost function, the objective function, and the loss function as 3 different things.

In our issues, we use these terms interchangeably, but for context:

• The loss function measures the error for a single training example.

• The cost function is the average of the loss function over the entire dataset. It gives an overall measure of the model’s performance.

• The objective function is the function that the optimization algorithm is trying to maximize/minimize, which can include the cost function and additional terms from regularization.

Technically speaking, an optimization algorithm optimizes the objective function, but it’s easier to think about the loss function (since it’s the distance between the actual and predicted results, which is the thing we want to minimize).

For a given dataset, an ideal optimization algorithm will navigate iteratively through a highly complex, multidimensional parameter space, adjusting each parameter to reach the lowest possible error.

Due to the mathematical nature of this issue, before continuing, it is highly recommended to have some background knowledge on multivariable calculus and an overall mathematical understanding of neural networks (as a layered structure that uses mathematical transformations to compute a relevant output).

Here are some issues to help with the intuition of neural networks:

• MLPs, Mathematically Explained (Beginner to Intermediate)

• Neural Networks, Explained (Beginner)

• Neural Network Learning (Beginner)