# CNN Backpropagation, Mathematically Explained

#41 Mar 10, 2025 Training

https://www.aibutsimple.com/p/cnn-backpropagation-mathematically-explained

This week’s issue is an extension of a past issue where we dived into a step-by-step math process of a convolutional neural network’s (CNN) forward pass. If you want to check it out, you can find it below.

- [CNN Forward Pass, Mathematically Explained (Intermediate)](https://www.aibutsimple.com/p/convolutional-neural-networks-explained-mathematically?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=cnn-backpropagation-mathematically-explained)

In this issue, we’ll be focusing on the backwards pass and CNN backpropagation, since it is a topic that is confusing to many learners.

#### Review of Backpropagation

Backpropagation is the process where we compute the gradients of the loss function with respect to each model parameter (weights and biases).

These gradients tell us how to update each parameter in order to minimize the loss using gradient descent. This is essentially how a neural network “learns” and gradually performs better by iterating through a dataset.

The key tool in backpropagation is the chain rule, which allows us to decompose the gradient of a composite function into a product of derivatives.

In CNNs, backpropagation is repeated layer by layer from the output layer back to the input, helping the network “learn” by updating the model parameters.

---

If you have a moment, clicking the ad below helps us a ton—and who knows, you might find something you’ll enjoy.

### Fact-based news without bias awaits. Make 1440 your choice today.

Overwhelmed by biased news? Cut through the clutter and get straight facts with your daily 1440 digest. From politics to sports, join millions who start their day informed.

[Sign up now!](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_humans&_bhiiv=opp_b17e63c9-bafc-4494-a919-f2f03f2cd21c_1b75ca79)

Also, before continuing, make sure you understand the basics of a CNN, such as how a convolution works and what hyperparameters such as stride and padding mean, along with some basic knowledge of calculus.

---

Let’s go through the mathematical process for a convolutional layer, step by step.

### Forward Pass

To understand the backwards pass and backpropagation, we need to first understand the forwards pass of a convolutional layer. Let’s say we have a 3×3 input matrix (X) and a 3×3 convolutional filter (W).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b09326a9-9327-401b-9258-4145ee7dae3c/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__1_.png)

Using the input matrix (X) and filter (W), we obtain the output feature map by performing a convolution operation between them.

- We’ll perform a convolution with padding 1 and a stride of 1, resulting in a 3×3 feature map.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/14c25813-5dbf-4d20-aa20-7dd5f9cfe3fb/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__6_.png)

After obtaining the feature map, we compute the pre-activation output (O) by adding a bias constant (b) to every element in the feature map.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95c63ea4-c858-4984-a02e-2ef4faf1f0bc/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___720_x_300_px_.png)

We can then obtain the activation output (A) by passing our output (O) through a ReLU activation function (ReLU = max(0, x)).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d384dc35-034d-4eb6-964d-d363f30064ef/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___720_x_300_px___1_.png)

Finally, we pass this output to the next layer, then the next, until we obtain the final prediction from the CNN.

Suppose the network outputs a scalar value after additional processing, and we compute some loss (L) using a function such as mean squared error.

### Backwards Pass

#### Overview of Required Gradients

In the convolutional layer used in the forward pass, we have two main parameters: the filter weights (W) and the bias constant (b).

So, in our backwards pass, we’ll compute the gradients of the loss (L) with respect to these parameters to update them.

However, we need to compute more than just two gradients. In our backpropagation step, we also have to compute the gradient of the loss (L) with respect to the input matrix (X) to continue to propagate the gradients backwards to the next layer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/74d7e7cf-3544-497c-84d6-7c8e7cd2ca7b/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___2_.png)

Additionally, we need to take into account our activation function (ReLU) to obtain the gradient of (L) with respect to our pre-activation output (O).

In total, given the gradient of loss with respect to our activation output (A) from the next layer, we need to compute four additional gradients.

But how do we actually obtain all of these gradients? We use our key tool: the chain rule for multivariable calculus.

#### Gradient with Respect to Pre-Activation Output (O)

Starting from the loss, we need to compute the gradient of the loss with respect to the pre-activation output (O) to be able to use it for other gradients within the layer itself (such as the gradient with respect to the filter weights (W)).

But to actually obtain this gradient, we use an initial gradient from the loss with respect to the activation output (A). Since A=ReLU(O), we use the chain rule to compute the gradient with respect to (O):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5100739c-1bda-4a44-b6dc-2df7cf126bb9/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px_.png)

Here, “⊙” denotes element-wise multiplication, also known as the Hadamard product.

The blue box highlights how we expand the gradient of the loss (L) with respect to the pre-activation output (O) by using the chain rule. Now, we have two gradient pieces to compute.

The first piece, the partial derivative of the loss (L) with respect to the activation output (A), is given from the previous layer in the backpropagation process—no computations needed here!

The second piece, the partial derivative of the activation output (A) with respect to our pre-activation output (O), is simply the derivative of the ReLU function itself.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/055ffd38-a8da-47a2-a8d8-164f7df474af/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_300_px_.png)

But what is ReLU’s derivative? ReLU’s derivative is obtained by taking the derivative of both pieces in the original ReLU piecewise function. The derivative turns out to be a very simple piecewise function: 1 if x is greater than 0 and 0 if x is less than 0.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7dcd1793-21d8-4e94-afe9-6d9488796a41/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb.png)

Note that the derivative of the ReLU function is technically undefined for x = 0, although in practice, many will use the value of 0 if x = 0.

#### Gradient with Respect to Filter Weights (W)

The next gradient we need to compute is the gradient of the loss (L) with respect to our filter weights (W).

Using the chain rule, we determine that this gradient involves our input matrix (X):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6de49002-cddf-4a73-8587-d64bff5d6bea/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___1_.png)

Above, the gradient with respect to our pre-activation output (A) was calculated in the last step, but what about the gradient of the output (O) with respect to the filter weights (W)?

- Imagine taking the partial derivative of W*X with respect to W. At this point, X becomes a constant, and using the power rule, W becomes 1.
- However, what’s different is that a convolution (represented using the symbol “∗”) is involved here, and thus a convolution ends up in the resulting gradient.

The resulting gradient with respect to the filter weights (W) is then simply taking the loss gradient with respect to the pre-activation output (O) and convolving it with the input matrix (X).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ff888fc9-a09c-4ea8-9861-d6b3f6afa4cd/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_300_px_.png)

We can also represent this using summation notation. For each element in (W), the gradient of the loss with respect to each element in the filter weights (W) is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bf418926-b1a4-4977-8e06-3cf840330b52/image.png)

#### Gradient with Respect to the Bias (b)

To calculate the gradient of the loss with respect to the bias (b), we also use the chain rule:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ac02c8e1-9541-4332-9faa-8c6d6df264d8/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___1_.png)

Here, the gradient of the pre-activation output (O) with respect to b is equal to 1, leaving just the gradient of (L) with respect to (O).

However, we’re still not done! In convolutional layers, the same bias is applied to multiple neurons (added to every element in a feature map).

Therefore, to compute the total gradient for that bias, you sum the individual gradients in that feature map.

Thus, for a layer with a shared bias (b) over multiple outputs, the gradient is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d9a93704-ccfb-4ae8-a368-181f14a0b9a4/image.png)

Here, the indices i and j iterate over the spatial dimensions (width and height) and potentially over the batch if processing multiple examples.

In simpler terms, the loss gradient with respect to the bias (b) is just the sum of all elements in the loss gradient with respect to the pre-activation output (O).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2211606e-e73d-47c8-82c0-36ff3d476f06/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___2_.png)

#### Backpropagating to the Previous Layer (Gradient with Respect to Input (X))

To propagate the error back to the previous layer (or the input), we need to reverse the forward convolution operation (but also continue to use the chain rule).

To do this, we convolve the loss gradient with respect to the output with a flipped version of the filter weights W (rotated 180 degrees).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/641b42c5-dd9d-4dcd-bd14-43e14376aecb/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___1_.png)

Above, W̃ denotes the filter (W) flipped both horizontally and vertically, which is equivalent to a 180-degree rotation. Here’s a quick visualization:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/876cf4cc-3410-4def-b331-5df10621ea3d/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__7_.png)

This step ensures that the gradients are aligned correctly with the input features. The resulting gradient of loss with respect to the input matrix (X) can be visualized like so:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cc8a4103-c025-423e-b3a1-6e4c343f7491/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_300_px___1_.png)

#### Updating Parameters

We’ve finally computed all the gradients needed to update the parameters for this convolutional layer and to backpropagate to the next layer!

To update our parameters, we can use the gradient descent update rule:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8099e258-6f94-4d16-b960-a3054bc00fd7/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___2_.png)

### Summary of CNN Backpropagation

In summary, in a convolutional neural network (CNN), backpropagation is used to adjust the filter weights (W) and biases (b) by computing gradients based on the loss.

- We can compute the gradient of the loss with respect to the pre-activation output (O) by multiplying the derivative of the ReLU activation function and the gradient of the loss with respect to the activation output (A).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f3c98afa-093a-4dfe-a27e-eb2459cefaad/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___1_.png)

The loss gradient with respect to the output (O) is the key gradient that will be used to calculate the other necessary gradients through the chain rule.

The gradient with respect to the bias (b) is simply the sum of the gradient with respect to the pre-activation output (O) across all positions, as the bias is shared.

The gradient with respect to the weights (W) is obtained by convolving the loss gradient with respect to (O) with the input (X), helping us update the filter values.

Finally, the gradient with respect to the input (X) is computed by convolving the loss gradient with respect to (O)​ with a 180-degree rotated version of the filter (W), letting the error propagate backwards.

Once we have ​the gradient with respect to the input (X), we pass it to the previous layer to continue the backpropagation process.

- This ensures that each layer in the network learns by adjusting its parameters to minimize the loss.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=cnn-backpropagation-mathematically-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=cnn-backpropagation-mathematically-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=cnn-backpropagation-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team