Issue #24 Training Edwin Dong
Optimization Algorithms in Deep Learning: Mathematically Explained
An optimization algorithm in deep learning is a method used to adjust the parameters (weights and biases) of a neural network to improve its predictions.
These algorithms are based on loss functions, functions used to measure how good or bad a given prediction is.
It calculates the difference between the network's prediction and the actual result.
If the output is 5, and the real value is 3, the loss is going to be greater than if the output was 4.
Some examples of loss functions
The goal of optimization is to reduce this error (or loss) as much as possible. Reducing this loss happens through shifting the numerical values of the parameters, and it makes the model’s predictions more accurate.
The optimization process stops when the algorithm reaches a point where the error can’t get any lower (or only gets lower very slowly). At this point, we say the model has converged. Ideally, this means the network is now as accurate as it can be on the given data.
