# Neural Network Backpropagation, Mathematically Explained

#82 Dec 22, 2025 Training

https://www.aibutsimple.com/p/neural-network-backpropagation-mathematically-explained

Imagine you were walking through a maze blindfolded. Once you take an incorrect turn, you feel a tap on your shoulder that’ll tell you “warmer” or “colder” depending on whether you’ve gotten closer or further away from the exit of the maze.

Although you still cannot see the maze, you use this feedback to understand how far you are from the exit, eventually finding the correct route out. This is effectively the process of backpropagation in neural networks.

As a neural network trains, it does not find out what the ideal weights should be to form a solution. Rather, all it learns is just how “wrong” it was and makes corrections to reduce the errors it makes.

Backpropagation is the calculus way to convert each error signal into information used to update the model. It does this for every parameter (weight or bias) in every layer, quantifying how much each weight contributed to the error.

### Getting Into the Math

To write this as a math equation, we represent the neural network as a differentiable function f:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/29196075-5185-4518-ad71-7225d3d9b547/image.png)

Above, x represents the input data fed into the neural network. For example, if the network was a language model, x might be a text prompt. But if the network were an image classifier, x would be an image.

W represents the weights or parameters in the neural network. Sometimes, people use the symbol θ (theta) to represent parameters, including biases.

Parameters are effectively the “dials” that tell the network how to transform the input to reach an output as close as possible to the ideal solution.

The function, f(x;W), is written in a notation that effectively says, “Take the input data x, and manipulate it with weights W.” The function f is the model itself that makes a prediction, ŷ. This prediction is the estimate the model makes, aiming to get as close to the true value, y, as possible.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/510008e6-1f8e-4df2-9ab3-e645862232ab/image.png)

Putting this together, the equation ŷ = f(x; W) says, “The predicted value is what the model outputs when you pass in the input data x through a network with weights W.”

---

### Can you scale without chaos?

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1a899050-5ea1-48ff-ad05-a45e6eead8d7/Beehiiv_Ad_Network_Campaign_-__CX_Retail__1200x600_graph__1_.jpg)](https://www.gladly.ai/cx-roi-calculator/?utm_campaign=parent-2025-09-interactive-content-roi-calculator-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_2b80e531-37f7-4c36-a23d-fa9656f09a80_2e98adb4)

It's peak season, so volume's about to spike. Most teams either hire temps (expensive) or burn out their people (worse). [See what smarter teams do](https://www.gladly.ai/cx-roi-calculator/?utm_campaign=parent-2025-09-interactive-content-roi-calculator-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_2b80e531-37f7-4c36-a23d-fa9656f09a80_2e98adb4): let AI handle predictable volume so your humans stay great.

[Check your options](https://www.gladly.ai/cx-roi-calculator/?utm_campaign=parent-2025-09-interactive-content-roi-calculator-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_2b80e531-37f7-4c36-a23d-fa9656f09a80_2e98adb4)

---

### Training: Minimizing the Objective

Training a neural network of this form means solving the equation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a6b83c24-bdfb-4b93-901c-504fc8152292/image.png)

To solve the equation, the objective is to minimize the value of L, given our network’s predictions and true values.

Essentially, what the statement is asking is what configuration of parameters we need in W to minimize L, the loss function.

L effectively tells you how far off the output of function f is from y, the ideal value, and minimizing it means that the model f is as accurate to the ideal as possible.

A neural network is built out of layers. Each layer performs two steps: a linear transformation and an activation. The first step is the linear transformation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fc9ce8e9-d201-48b5-bbac-91136697f59d/image.png)

This is a matrix multiplication of the weights and the input to the layer, a, followed by the addition of a bias, b. Then, this output z is passed into a non-linear activation function, typically a ReLU variant or sigmoid.

This is reflected in the second step:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6154194b-ce0d-4996-8c15-e873ea339aa9/image.png)

Here:

1. z(l) is the raw weighted sum
2. σ is a non-linear activation function
3. a(l) is the activated output

Activation functions introduce non-linearity, allowing the network to learn complex patterns, rather than becoming one large linear equation. Here are some common activation functions:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/32b9aafb-2743-447d-9bf1-e8dcedfbbb22/Activation_Functions__1_.png)

The diagram below demonstrates the mathematics of a single neuron inside a neural network.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/78b23c3c-1e5f-4734-9632-347745f39289/image.png)

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-network-backpropagation-mathematically-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-network-backpropagation-mathematically-explained)

---

Without these nonlinear activations, a deep network becomes equivalent to a single linear transformation, no matter how many layers it has. The forward pass for a multi-layer network looks like:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c3d3e8d9-c735-4b12-a70e-93bc76c7219c/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9a436be4-2125-4c95-ac6e-9aab263887cd/image.png)

The key idea in this forward pass system is that every layer transforms the previous layer’s activations. All the intermediate values (the z’s and the a’s) are stored because they will be used later in backpropagation.

Backpropagation is important to determine how much every weight contributed to the network, and it does so by applying the chain rule of calculus through the entire network.

To understand why we utilize this structure, it helps to first see how gradients would be computed without it.

The central question in solving neural network equations is finding out how to compute ∂L/∂W efficiently for hundreds of thousands, millions, or occasionally billions of parameters. Every weight affects the loss indirectly through several intermediate transformations. The chain rule tells us:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fd1b1236-203e-4e98-8f91-ec15779029f5/image.png)

This is the fundamental basis for backpropagation. Computing gradients individually is impossibly slow. Instead, backpropagation introduces the delta (δ) value:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/32388d76-e424-4078-921e-9ed28370e82e/image.png)

This delta finds the corresponding change in loss of layer L, based on how much output z changes, even if only slightly. With deltas, backpropagation becomes efficient. The output layer states:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ed2e48c3-cf71-4baa-8420-6d00ac9e2590/image.png)

Earlier layers:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d4e86913-2ab5-41f4-ae1e-e4655e807008/image.png)

This is the backwards flow of error.

Weight gradients then follow immediately. But once we have the deltas, updating weights becomes simpler:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/47904af2-f841-43d5-ad43-1230a01bf5af/image.png)

Weights are then updated using gradient descent throughout the entire network.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/06f25cf4-b361-4efb-8dcd-d82a5a28bd6f/image.png)

Here, α is the learning rate, W(l) represents the weight matrix at a given layer l, and ∂L/∂W(l) is the gradient of the loss function L with respect to the weight matrix. The gradient is a vector containing all the partial derivatives of the loss with respect to each entry in the weight matrix.

Imagine the gradient as a vector in a high-dimensional loss landscape. The gradient points in the direction of steepest ascent, so subtracting the gradient moves the weights in the direction of the steepest descent, reducing the loss of the network.

This update happens across all layers, repeating millions of times throughout training.

Backpropagation turns error signals at the output into precise corrections across the entire network. It is one of the most important algorithms in modern AI, the silent engine behind every neural network, from basic classifiers to large language models.

If the model is the explorer in the maze from our earlier analogy, backpropagation is the shoulder tap that tells it how to correct its path, step by step, until it eventually finds its way.

### A Worked Example

To make the mathematics of backpropagation clearer, we will demonstrate an example of a backward pass in a neural network. To begin, we define our input x:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7af57b82-f33d-4b2b-925b-5381e7339aff/image.png)

The hidden layer weights (W(1) ) and the biases (b(1) ) are initialized randomly to:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/997eda3e-424a-48b3-b4f1-d2c0e5d9bde2/image.png)

The hidden pre-activation equation (z(1)) is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/022d5d6a-ccee-4cad-b092-7a16c6d25cb8/image.png)

The hidden activation vector (a(1)) is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/01ab7437-7781-4e9b-88eb-9446bc33a779/image.png)

The output layer weights (W(2)) are also initialized randomly. We’re going to use:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6d47a559-5dd0-4dca-b20e-f8a57b3f2237/image.png)

The output is computed using the same pre-activation (z(2)) to activation (a(2) ) transformation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/54403879-22cc-458e-9726-e565b9cd0d0d/image.png)

To compare this value and compute the loss, let’s say we have a target value of 1:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ebedfec4-9f35-47ab-865c-ae29b57d4a97/image.png)

With this, we can calculate the loss by using the mean-squared error:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8650ec4f-6ee4-4a77-93ed-52dde5b9872d/image.png)

Now, it’s time for the backward pass, so we move backwards. We start by calculating the output layer delta (δ(2)):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6f1a0d9f-259b-4717-86eb-5b8e4ec82e6a/image.png)

This value we have just calculated is a measure of how incorrect the network was at its output. We can then move to finding this output layer’s weight gradients (∂L/∂W(2) ).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ef639e1d-26af-4bb7-b97f-3fb08bd3b2e1/image.png)

For the same δ, we have two weight values. Now, we can move to the hidden layer, calculating the hidden layer delta like so:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a6ff85f3-fe04-45a1-8407-e9b34a86524c/image.png)

And finally, we compute the gradient of the weight with respect to the hidden layer (∂L/∂W(1) ):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/228f49d7-c5c1-42f2-959b-c9f2aba94c77/image.png)

Using this, we can update the weight matrices at each layer using our gradients of the loss with respect to the weight matrix at that layer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/52b5b3bf-56f9-474e-b6eb-aa810483d80a/image.png)

And that is exactly how neural networks iteratively tune their parameters using calculus to “learn.”

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-network-backpropagation-mathematically-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-network-backpropagation-mathematically-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neural-network-backpropagation-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team