Neural Network Backpropagation, Mathematically Explained

Imagine you were walking through a maze blindfolded. Once you take an incorrect turn, you feel a tap on your shoulder that’ll tell you “warmer” or “colder” depending on whether you’ve gotten closer or further away from the exit of the maze.

Although you still cannot see the maze, you use this feedback to understand how far you are from the exit, eventually finding the correct route out. This is effectively the process of backpropagation in neural networks.

As a neural network trains, it does not find out what the ideal weights should be to form a solution. Rather, all it learns is just how “wrong” it was and makes corrections to reduce the errors it makes.

Backpropagation is the calculus way to convert each error signal into information used to update the model. It does this for every parameter (weight or bias) in every layer, quantifying how much each weight contributed to the error.