Multilayer Perceptrons: The Math Behind Deep Learning

In the field of deep learning, we solve machine learning problems while using Artificial Neural Networks (ANNs).

One of the most common first learning steps in deep learning is to learn the Multilayer Perceptron (MLP), which is a specific type of Artificial Neural Network (ANN).

An MLP is a type of feedforward artificial neural network, composed of at least three layers: an input layer, one or more hidden layers, and an output layer.

• If you are to say to someone that you understand deep learning, it is essential to understand how MLPs work. They are the most common and basic when it comes to technicalities.

In our previous issues, we’ve covered a lot about propagation, nodes, activation functions, and gradients of MLPs.

• In this issue, we’re going to sum up the process and go into the math of it.

Before continuing, it is highly recommended to have some basic knowledge of linear algebra and calculus.

Mathematics of MLPs

From a mathematical perspective, MLPs are just a set of functions where each layer applies transformations to an input to produce a corresponding output.

Starting with the structure, each MLP has at least an input layer, one or more hidden layers, and an output layer.

If you have a neural network with n layers, counting both the input and output as a layer, you will have n-1 weight matrices.

• Biases work the same way; if you have n layers, you will have n-1 bias vectors.

For instance, if you have 3 layers (one input, one hidden, and one output), you will have one weight matrix connecting the input to the hidden layer, and another connecting the hidden to the output layer.

The dimensions of the weight matrices are determined by the number of neurons in the incoming layer along with the number of neurons in the outgoing layer.

• Let’s take the weight matrix connecting the input to the hidden layer. If the input layer has 20 neurons, and the hidden layer has 40 neurons, then the weight matrix will have a size of 40 × 20 (rows x columns).

Here, the superscript is acting as an index for the weight matrix. Since this goes from input to hidden, it is the first. Then, the second would be the hidden to the next layer.

• The subscript denotes the column and row for easy indexing. In later examples, we will remove the comma.