Recurrent Neural Networks, Explained Mathematically

A Recurrent Neural Network (RNN) is a special type of neural network designed to handle sequential data.

• Sequential data is any type of data where the order of the data is important. These can be words, sentences, or time-series data.

At its core, an RNN is trained to take sequential data as an input, and using its hidden state (which acts like its working memory), it transforms the data into a specific output of sequential data.

Unlike feedforward neural networks, RNNs have connections that form cycles, allowing information to be remembered over time.

We’ve gone over RNNs extensively in previous issues; you can read them to learn the theory behind RNNs:

• Recurrent Neural Networks, Simplified (Beginner)

• Modern Recurrent Neural Network Architectures (Intermediate)

This week, we’re going into the mathematics of simple RNNs, so a base understanding of the concept of an RNN needs to be learned.

Also, for new subscribers or beginners, knowing a bit of context on the mathematical processes of neural networks would be good as well, along with a bit of basic linear algebra (matrix multiplication, vectors, etc.) and calculus.

You can find a math explanation for a simple Multi-Layer Perceptron (MLP), one of the simplest forms of ANNs, here.