# MLP Learning Process: The Math Behind Deep Learning

#18 Sep 23, 2024 Foundation

https://www.aibutsimple.com/p/mlp-learning-process-math-behind-deep-learning

Last week, we discussed the forward pass of a Multi-layer Perceptron (MLP). We went through pre-activation, activation, and layer-to-layer matrix math using the iris dataset.

- If you want to read it, you can find it [here](https://www.aibutsimple.com/p/multilayer-perceptrons-math-behind-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mlp-learning-process-the-math-behind-deep-learning)

This week’s letter is a continuation of the MLP learning process, notably the backwards pass and updates.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1924d831-6042-4186-a7c5-1847e4268f84/image.png)

We’ll continue to use the iris dataset for the examples.

To summarize, the entire training process of the MLP goes like this:

- Randomly initialize weights and biases
- Forward Propagation
- Compute Loss
- Backward Propagation
  - Compute the gradients of the loss with respect to weights and biases
- Update Weights and Biases
  - Use an optimization algorithm like Gradient Descent to update the weights and biases
- Repeat

We’ll be focusing more on backpropagation and parameter updates in more detail in this issue.

But before we get into the gradient calculations, let’s review some basic multivariable calculus:

A partial derivative (∂y/∂x) is a derivative of f with respect to some variable x.

For example, if f(x,y)=x2 + y2, (∂f/∂x)=2x, because y2 is simply a constant with respect to x.

- Make sure you understand how partial derivatives work and how the chain rule for multivariable functions works before proceeding.

Let’s use the same sample we used last time:

- Sepal Length (x1​): 5.1 cm
- Sepal Width (x2): 3.5 cm
- Petal Length (x3​): 1.4 cm
- Petal Width (x4​): 0.2 cm

This time, we’ll include a label: Iris Setosa (Class 0)

The true label vector y is one-hot encoded to be:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6280c797-8b7f-411a-941d-42aa23e05364/image.png)

We’ll use the same 4 initially random weight matrices and biases, shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a30c2f3b-1b5d-426a-9092-cab5d87a8fb3/ql_05d5e6ba0de3c0471cc59b0505de2cbe_l3.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd2b54f9-b5b0-415c-86a4-7f9fde4f20d7/ql_33a684804edf92aa94a5c63d4f8b568b_l3.png)

W(2) is size 3×6 and b(2) is size 3×1; they are not pictured.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/publication/logo/38317951-cfc3-4ab4-9700-9deaced49ef0/Opaque_Youtube_Profile_300_x_300.png)

Sponsored

AI ConfidentialAccelerating AI Innovation With Data Privacy & Sovereignty

Subscribe

The model will remain the same, with 4 neurons in the input layer (one per feature), 6 neurons in the hidden layer, and 3 neurons in the output layer (one per class):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/45c4dfb2-34de-4517-9701-44ee910730b8/Copy_of_Copy_of_Add_a_heading__24_.png)

Continuing from last week’s example, let’s assume the model comes up with this output vector:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7aa9cbcc-642a-4196-b9a8-5cc8e549fee0/image.png)

### Error Term for Output Layer

We start off by computing the error at the output layer, noted as δ(2) (delta is the error, the superscript denotes that it is the 2nd error as we are working backwards).

The error term δ(2) at the output layer represents how much the network's output deviates from the true label. It's calculated as the derivative of the loss function with respect to the pre-activation input z(2) at the output layer.

Since this is a multi-class classification problem using the softmax activation, we use cross-entropy loss:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c8e21d93-3e4a-485a-b04a-cb2fe0f258bc/image.png)

- yk is the one hot encoded true label.
- ak(2) is the predicted probability for class k.

The derivative of the loss L with respect to zk(2) is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/66cbe049-3a25-40f7-97ee-493fa4f51822/image.png)

This formula takes this form because the derivative of the softmax function (which is the activation function) combined with the cross-entropy loss simplifies to this expression.

You might be wondering why we use the derivative of the activation function.

- We use it simply because when we backpropagate from one layer to the previous, we need to account for the activation functions used.
- The error at a neuron depends not only on the errors in the subsequent layer but also on how sensitive the neuron's activation is to its input.

So, the error term at the output layer is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18e8894d-94e1-416e-a4dd-da276e29bccf/image.png)

Here, a(2) represents the activation vector at the output layer, and y is the one hot encoded input.

The k’s are removed and replaced with the entire vectors.

The error term ends up being computed as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9903b412-041c-4ad3-ade0-c7a2cda3bc71/image.png)

Since we have the error from the output layer, we can now use it to calculate the derivative of loss with respect to weights and biases at this layer (W(2) and b(2) ).

### Compute Gradients at Output Layer

We need to compute the derivative of loss with respect to W(2) to update the weights from the hidden layer to the output layer using an optimization method such as gradient descent.

The derivative can be unpacked using the chain rule:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b2e7c741-d7b8-4011-a671-7ad8ec393c57/image.png)

Remember, z(2) is calculated as shown below. The weight matrix has an independent variable a(1) while the bias does not (like a constant).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8a406af4-c8a2-4796-99ea-f3fae92ab5c4/image.png)

Then, we compute the derivative of loss with respect to W(2) by doing a couple of variable switches:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2ff84566-8277-42ce-b028-566398653b72/Input_to_Hidden__7_.png)

The derivative of z(2) with respect to W(2) turns out to be a(1), and the derivative of L with respect to z(2) is δ(2) , proven earlier.

The transpose on the activation vector is for matrix multiplication purposes: the error term (δ(2) ) is size 3 × 1, while the activation vector is transposed from 6 × 1 to 1 × 6, allowing the matrix produced when multiplied to have the size of 3 × 6 (which matches the size of the weight matrix W(2) ).

Let’s numerically compute the derivative of the loss with respect to weights:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3623433d-dd54-43fe-ab25-0397b2b76631/Input_to_Hidden__600_x_250_px___1_.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/893134d8-6577-40f2-a9ee-a3a3c58c64b7/image.png)

This is the pattern for matrix multiplication. Note that the columns in the first matrix must match the rows in the second matrix (3 × 1 and 1 × 6) for the multiplication to be applicable.

We end up with this matrix of values:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/37ba7775-b4f8-4776-b662-e2c57971d446/image.png)

Next, we’ll compute the derivative of loss with respect to biases of the output layer, b(2).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9bd92444-cd82-4842-9812-885b257c9ea5/Input_to_Hidden__6_.png)

Using the chain rule, we can unpack the derivative into 2 parts. The derivative of loss with respect to z(2) is δ(2) , as shown prior, and the derivative of z(2) with respect to biases is 1, as there is no additional variable multiplying the biases.

At this point, we have the derivatives needed to update the weight matrix W(2) and the bias vector b(2).

Continuing with the backpropagation process, we should also update the weight matrix from the input to the hidden layer (W(1)) and the bias vector for the hidden layer (b(1)).

### Backpropagate Error to Hidden Layer

To perform these updates, we need to backpropagate the error to the hidden layer.

δ(1) represents the error term for the hidden layer. It quantifies how much each neuron in the hidden layer contributed to the overall error in the output.

Let’s compute the derivative of loss with respect to z(1) to modify the weights and biases later.

Using the chain rule, we can obtain this gradient:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/49c76c9d-1b91-4573-a440-f7d64f35e51f/Input_to_Hidden__9_.png)

The derivative of loss with respect to the output vector a(2) multiplied with the derivative of a(2) with respect to the pre-activation z(2) can be simplified to the derivative of loss with respect to z(2) , which is equal to the error term at the output layer δ(2).

Based on the equation of z(2) , taking the derivative of z(2) with respect to the hidden layer activation vector a(1) is W(2).

The derivative of the hidden layer activation vector a(1) with respect to its pre-activation z(1) will involve the sigmoid function, as it is the activation function we chose.

We obtain the error term for the hidden layer by arranging the matrices in a specific way and transposing W(2) for multiplication. We introduce the [Hadamard](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mlp-learning-process-the-math-behind-deep-learning)(element-wise) product for calculating the hidden layer error term because the derivative of the activation function is applied element-wise to each neuron in the layer.

Assume this is W(2) after being transposed:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6d67e489-322d-4419-b5c1-c9e5aa7f01da/image.png)

Using the δ(2) calculated prior, we can compute it together with W(2) transpose. I’ll call this block 1.

The size of the product will be 6 × 1, as W(2) transpose has a size of 6 × 3, and δ(2) has a size of 3 × 1.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/53413aac-ef32-4db0-83fe-f6be0c614a3f/image.png)

Here, we used the dot product for calculations.

Moving on, the derivative of the hidden layer activation vector a(1) with respect to its pre-activation z(1) simplifies to:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2ea44535-eec4-4d45-8204-e80828021e46/image.png)

I’ll call this block 2.

This will have a size of 6 × 1 as well, as a(1) has size 6 × 1. This makes it compatible to use the Hadamard product to calculate the error term.

For simplicity sake, assume the Hadamard product of block 1 and block 2 (which is the error term at the hidden layer δ(1) ) computes to be:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18088a1e-7175-436f-958e-e5ba102fedc9/image.png)

After obtaining the error term, we can finally calculate the derivative of loss with respect to weights and biases of the hidden layer.

### Compute gradients at hidden layer

Let’s compute the derivative of loss with respect to W(1) , the weights of the input layer to the hidden layer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/080af501-3480-42c8-9d47-c63bf1e4eaaa/Input_to_Hidden__10_.png)

It ends up simplifying to the hidden layer error term multiplied by the input transposed.

Let’s compute it numerically:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/254a93c3-b354-4c4c-832e-2b5a891129b1/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/979020b8-c56a-4dc4-864e-e2dfb06f26e4/image.png)

It ends up being size 6 × 4, since δ(1) has size 6 × 1 and xT has size 1 × 4.

We can also compute the derivative of loss with respect to the hidden layer biases b(1) using the chain rule:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/df18ce4f-9146-421f-8b6d-092efa968439/image.png)

Using the same process for the derivative of loss with respect to the output layer biases, the derivative of loss with respect to z(1) is δ(1), and it is multiplied with the derivative of z(1) with respect to b(1) , which is 1.

### Parameter Updates

We now have all of our gradients needed to update all of our weight matrices (W(1), W(2) ) and all of our bias vectors (b(1), b(2)).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3d9a06e6-eaac-481e-9e14-b5bf9ec473ed/image.png)

We can use gradient descent to update the weights by subtracting the derivative of loss with respect to a certain parameter, multiplied by the learning rate α, shown above.

If we subtract the gradient (multiplied by a learning rate), we will always be moving in a direction closer to a minimum.

To understand more of why gradient descent converges the parameters like weights and biases towards the minimum, you can check out our issue [here](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mlp-learning-process-the-math-behind-deep-learning).

We can do a sample calculation of the hidden layer weight matrix:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/52deeb37-d89d-499c-b6c6-453ce418d9c1/image.png)

Then compute the numerical values:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/25fac5cc-53d7-4e46-ac4e-eee0f595c2ba/image.png)

We repeat this for the rest of the parameters.

After computing the new weight matrices and bias vectors, we are done with the backwards pass for a training example or a batch.

We can continue to do forward passes and backward passes in epochs to train the neural network.

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=mlp-learning-process-the-math-behind-deep-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team