# Multilayer Perceptrons: The Math Behind Deep Learning

#17 Sep 16, 2024 Foundation

https://www.aibutsimple.com/p/multilayer-perceptrons-math-behind-deep-learning

In the field of deep learning, we solve machine learning problems while using Artificial Neural Networks (ANNs).

One of the most common first learning steps in deep learning is to learn the Multilayer Perceptron (MLP), which is a specific type of Artificial Neural Network (ANN).

An MLP is a type of feedforward artificial neural network, composed of at least three layers: an input layer, one or more hidden layers, and an output layer.

- If you are to say to someone that you understand deep learning, it is essential to understand how MLPs work. They are the most common and basic when it comes to technicalities.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/929c4a11-484b-4536-9ee7-35c48c6ed7eb/Add_a_heading.png)

In our previous issues, we’ve covered a lot about propagation, nodes, activation functions, and gradients of MLPs.

- In this issue, we’re going to sum up the process and go into the math of it.

Before continuing, it is highly recommended to have some basic knowledge of linear algebra and calculus.

### Mathematics of MLPs

---

From a mathematical perspective, MLPs are just a set of functions where each layer applies transformations to an input to produce a corresponding output.

Starting with the structure, each MLP has at least an input layer, one or more hidden layers, and an output layer.

If you have a neural network with n layers, counting both the input and output as a layer, you will have n-1 weight matrices.

- Biases work the same way; if you have n layers, you will have n-1 bias vectors.

For instance, if you have 3 layers (one input, one hidden, and one output), you will have one weight matrix connecting the input to the hidden layer, and another connecting the hidden to the output layer.

The dimensions of the weight matrices are determined by the number of neurons in the incoming layer along with the number of neurons in the outgoing layer.

- Let’s take the weight matrix connecting the input to the hidden layer. If the input layer has 20 neurons, and the hidden layer has 40 neurons, then the weight matrix will have a size of 40 × 20 (rows x columns).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/31ed3eec-422b-4a41-b4c9-83a988ab7480/ql_adb1e48ae3311b77816b98c1c9a4cfb7_l3.png)

Here, the superscript is acting as an index for the weight matrix. Since this goes from input to hidden, it is the first. Then, the second would be the hidden to the next layer.

- The subscript denotes the column and row for easy indexing. In later examples, we will remove the comma.

However, in MLPs, biases work a little differently as they only have one column. An input layer will not have a bias vector, they only start from the first hidden layer.

It will take the dimension of the number of nodes of the hidden layer x 1, and given a hidden layer with 5 nodes, it will look like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/04a74d97-30c2-4998-9754-1d16bc3f5795/ql_3fdff63e7677c8162a0240b5cfc469af_l3.png)

#### Full Example

We’ll do a full example with the Iris dataset now.

The Iris dataset is a classic dataset used in machine learning for classification tasks. It contains 150 samples of iris flowers from three species:

- Iris Setosa
- Iris Versicolor
- Iris Virginica

Each sample has four features:

- Sepal Length (cm)
- Sepal Width (cm)
- Petal Length (cm)
- Petal Width (cm)

So the input matrix has a size of 150 × 4, and we will denote it as x.

The MLP we will be using will consist of one input layer, one hidden layer, and one output layer.

The input layer will have 4 neurons, allocating one neuron per feature (column), the hidden layer will have 6 neurons from arbitrary choice, and the output layer will have 3 neurons—one for each class.

The network looks like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/45c4dfb2-34de-4517-9701-44ee910730b8/Copy_of_Copy_of_Add_a_heading__24_.png)

The labels on the bottom correspond to the weights and biases the network has.

The weight matrices and biases look like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/af7ab8be-80b0-4b6f-876a-4429168d5538/Input_to_Hidden.png)

Here, the comma in the subscript is removed, so 11 represents 1, 1.

#### Forward Pass

Let’s go through a simple process of forward propagation. Suppose we have an iris flower with the following features:

- Sepal Length (x1​): 5.1 cm
- Sepal Width (x2): 3.5 cm
- Petal Length (x3​): 1.4 cm
- Petal Width (x4​): 0.2 cm

This would represent one row in the 150 examples.

First, we would compute the “pre-activation” (denoted as z) for the hidden layer; simply a matrix that has the hidden layer’s mathematical operations done on it before applying an activation function.

The formula shown below is for z(1), the vector being transformed from the input to the hidden layer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/51482c3d-29d5-4883-8e8a-a68cb1b86222/ql_e87fb535beaef5b4f5986c4b3ad62ace_l3.png)

This can be unpacked into this formula to calculate each individual entry in the matrix:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/570b0527-119f-40a3-977b-d41777d65f9a/ql_ad44c10d8a316c2d15d1c8a58220a0d0_l3.png)

Let’s go over a sample calculation for z1.

Since weights and biases are initialized randomly, let’s take this matrix of weights and this bias vector:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a30c2f3b-1b5d-426a-9092-cab5d87a8fb3/ql_05d5e6ba0de3c0471cc59b0505de2cbe_l3.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd2b54f9-b5b0-415c-86a4-7f9fde4f20d7/ql_33a684804edf92aa94a5c63d4f8b568b_l3.png)

Then, the calculation for z1 is as follows: we take each element of the first row of the weight matrix and multiply it with the corresponding input (mentioned prior: sepal length, width, etc.). We add the first row of the bias vector to this weighted sum to obtain z1.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bc25c7e1-52e0-4713-b7b9-eb4940fcc677/Input_to_Hidden__1_.png)

We should repeat these steps for z2 to z6:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/560e197a-3367-4c3e-8986-d9587ee056a3/ql_0e3f8f0b01acfb0d485b801a3e2e7b32_l3.png)

After computing the z(1) vector, we’ll feed it through an activation function. For this case, let’s use the sigmoid activation function. We will denote the activation vector as a.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c762c299-c0b6-4536-9374-84f2906896c2/Copy_of_Input_to_Hidden.png)

Let’s do a sample computation for a1:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/decb78d6-2b4b-49e2-ae95-23b183d6fd02/image.png)

Next, we’ll compute the entire activation vector using the formula:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/86a7151d-1c70-441c-a21f-2a6796b20ec3/image.png)

We have just completed the process for passing data from the input layer to the hidden layer. Let’s continue with the hidden to the output layer.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/35fd80ce-173b-4f5f-9cce-818789af7b06/ql_17cb62764eb5d9a82c39826b6856d560_l3.png)

Note that if we had more than one hidden layer, we could go on with z(3) and a(2) and so on.

Using randomly initialized weights and biases in the hidden to output weight matrix and the output bias vector, assume we end up with z(2) like so:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd230df3-4fec-450d-a269-5fc0ead93dcb/image.png)

Since this vector corresponds to our output layer, and it is a classification problem, we’ll use the softmax activation function to get a probability distribution with values ranging from 0 to 1—the probability the training sample belongs to a certain class.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/328dd0ea-7cc5-4d75-b38a-e3e9d1ad36ad/Copy_of_Input_to_Hidden__1_.png)

Every element in the second activation vector (a(2)) is calculated as shown above.

We end up with:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7aa9cbcc-642a-4196-b9a8-5cc8e549fee0/image.png)

Each one of these entries shows the probability of the training sample belonging to a certain class. We can convert this to a percent:

- Class 1 (Setosa): 43.23%
- Class 2 (Versicolor): 33.72%
- Class 3 (Virginica): 23.05%

The neural network will take the class with the highest probability for its prediction, choosing class 1 (Setosa) accordingly.

The forward propagation process for one training sample is now complete. We can repeat this with as many training samples as given.

---

Give yourself a pat on the back if you made it this far! Now you understand the math background behind the forward pass.

We won’t cover the backwards pass in this issue, so if you want to see the mathematical interpretations of the backwards pass, backpropagation, and the learning process, check back next week for the next issue!

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, Inquiries, advertising? Send us an email at edwin@aibutsimple.com.

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=multilayer-perceptrons-the-math-behind-deep-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team