Issue #22 Architecture Edwin Dong
Recurrent Neural Networks, Explained Mathematically
A Recurrent Neural Network (RNN) is a special type of neural network designed to handle sequential data.
Sequential data is any type of data where the order of the data is important. These can be words, sentences, or time-series data.
At its core, an RNN is trained to take sequential data as an input, and using its hidden state (which acts like its working memory), it transforms the data into a specific output of sequential data.
Unlike feedforward neural networks, RNNs have connections that form cycles, allowing information to be remembered over time.
We’ve gone over RNNs extensively in previous issues; you can read them to learn the theory behind RNNs:
This week, we’re going into the mathematics of simple RNNs, so a base understanding of the concept of an RNN needs to be learned.
Also, for new subscribers or beginners, knowing a bit of context on the mathematical processes of neural networks would be good as well, along with a bit of basic linear algebra (matrix multiplication, vectors, etc.) and calculus.
You can find a math explanation for a simple Multi-Layer Perceptron (MLP), one of the simplest forms of ANNs, here.
Before jumping into the math, we’re proud to say that we’ve partnered with Bullseye Trades for this sponsored segment!
Free Daily Trade Alerts: Expert Insights at Your Fingertips
Master the market in 5 minutes per day
Hot stock alerts sent directly to your phone
150,000+ active subscribers and growing fast!
Mathematical Formulation
This week, we’re going over a standard RNN for text generation. These types of RNNs will be fed data and be asked to produce the next words to be able to generate text.
Let’s start with an example using a small phrase: “Hello World”.
We’ll use the below vocabulary for the model. This means that the model only has access to these words in training. We can call these individual words tokens.
<START>
Hello
World
<END>
Here, you might be wondering what the words 1 and 4 are. RNNs typically use start and end tokens to denote where a sentence starts and where a sentence ends.
Each word is represented as a one-hot vector:
To train a RNN, we need to feed it a data point: an input sequence. Using the phrase “Hello World” works a little differently using start and end tokens.
