# A Simple Guide To RNNs

#43 Mar 24, 2025 Architecture

https://www.aibutsimple.com/p/a-simple-guide-to-rnns

A recurrent neural network (RNN) is a deep learning model that is trained to take a sequential data input and transform it into some sequential data output.

- Sequential data is any type of data where the order of the data is important. These can be words, sentences, or time-series data.

RNNs are widely used in natural language processing (NLP) tasks, including language translation and speech recognition.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6e3e6f01-07bf-4e57-a904-d5f940ebb6ff/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___1_.png)

The key feature of RNNs is their ability to maintain a hidden state that captures information from previous time steps, effectively creating a form of memory. It is the middle portion of the network shown above (denoted ht).

This hidden state is updated at each time step (which we denote as t) as the network processes new input data, enabling the network to learn from both the current input and the historical context.

In terms of how RNNs work mathematically, they have two main equations to compute: the hidden state ht and the output yt.

For each timestep t, the hidden state ht is expressed as follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3b7cf5da-1728-44cf-82fc-311925598d80/image.png)

Where Wx represents the input to hidden weight matrix, Wh represents the hidden to hidden weight matrix, bh is the hidden bias, f1 is some activation function (usually tanh or sigmoid), and xt is the input.

For each timestep t, the output yt is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/20c078af-91fe-448a-84d5-8ac1e4dc41a9/image.png)

Here, Wy represents the hidden to output weight matrix, by represents the output bias, and f2 is an activation function.

The weight matrices and biases included in both the hidden state and the output are shared temporally.

- This means that the same set of weights is used to compute the hidden state at every time step, which is how RNNs can process sequences of arbitrary length while learning to recognize patterns over time.

### Activation Functions

The most common activation functions used in RNNs are the sigmoid, tanh, and ReLU functions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a03f77ba-98b2-49dd-a43d-f4af9d9e79c3/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x___1_.png)

A surprising thing to note is that standard RNNs use the hyperbolic tangent/sigmoid function very often (even more than ReLU), as using other activations such as ReLU may create such large numbers in certain conditions that it can lead to overflow.

Properly placed tanh/sigmoid functions can avoid this. In the worst case, using tanh/sigmoid activations will produce saturation and vanishing gradients, which is better than overflow (which causes your model to break down).

- Want to learn more about activation functions? Check out our past issue [here](https://www.aibutsimple.com/p/all-about-activation-functions?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-simple-guide-to-rnns).

### Types of simple RNNs

There are many different types of simple RNNs used for different applications in NLP and speech recognition. Here are some common configurations:

One-to-one RNNs have an input and output sequence length of 1 (Tx=Ty=1). They are like traditional neural networks.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/458bc7f4-3e52-4ca6-8b5f-16a130f60d83/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x.png)

One-to-many RNNs have an input sequence length of 1 but a longer output sequence length (Tx=1,Ty>1). An example use case of this configuration is music generation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e8cf07a8-f503-4261-9f6a-47165e4373e7/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x__1_.png)

Many-to-one RNNs have longer input sequences but an output sequence length of 1 (Tx>1,Ty=1). This configuration of RNN is used in sentiment classification/analysis.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5231b226-8d24-442f-9fc7-9b1723bd8400/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x__4_.png)

Many-to-many RNNs have long sequences in the input and output, while the length of the sequences can be equal or not equal (Tx=Ty OR Tx≠Ty).

The Tx=Ty configuration is used in something like named entity recognition (NER), where the goal is to classify entities (like people, locations, and organizations) in a sentence.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/696dd89d-b6dc-4401-ae6a-f1e5accdad6c/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x__6_.png)

The Tx≠Ty configuration is used for machine translation, as the translated sentence can vary in length—it’s usually not the same length as the input sentence.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/10dc4316-3821-4f7a-ba30-49138028e1c3/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x__7_.png)

### Loss and Backpropagation

In the case of a recurrent neural network, the loss function L of all time steps is the sum of the loss at every time step as follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7408496d-63b8-4d5d-b3c7-e0a569fa590c/image.png)

​ For example, using the cross-entropy loss, it would look something like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3eabb389-f09c-4e81-a17a-0e996ee3c1bc/image.png)

RNNs learn through backpropagation and optimization algorithms, just like any other neural network.

Backpropagation is a method to update the weights in the neural network by taking into account the actual output and the desired output. The derivative with respect to each parameter (weights and biases) is computed using the chain rule.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0d6fa016-5a1e-4ac7-b26b-609bd1d814ca/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_headin.png)

We use these partial derivatives in the optimization algorithm’s update rule. For instance, with gradient descent, we simply to subtract the negative gradient.

The difference between backpropagation in standard ANNs and RNNs is that backpropagation is done at each point in time (backpropagation through time (BPTT)) for RNNs. At the final timestep T, the derivative of the loss L with respect to weight matrix W is expressed as follows:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fbb854cb-3dd6-4fdc-8cde-12c348a180d6/image.png)

During BPTT, you compute the gradient of the loss with respect to the weights by applying the chain rule backward through time, accumulating gradients as you go.

### Vanishing and Exploding Gradients

One of the biggest problems with RNNs is the vanishing and exploding gradient phenomenon.

The reason why they occur is due to the multiplicative nature of the backward pass (and a long sequence length), causing gradients that exponentially decrease or increase with respect to the number of layers.

To illustrate a vanishing gradient, imagine a neural network with some magic layers that reduce the backpropagation gradient signal by 30% each time (usually due to tanh or sigmoid activation). After just 10 layers, the signal is multiplied by 0.710 = 0.028, which is around 35 times less than the first layer!

- The same example can be extended to exploding gradients, where gradients can be much larger than the first layer during the backward pass.

### Advantages and Disadvantages

A typical RNN architecture has its advantages and disadvantages. They made it possible to process an input of any length, share weights across time, and implement a sort of memory.

However, compared to newer models, they are slow and computationally intensive and don’t perform well with longer sequences (the RNN will end up “losing” important information).

- When long sequences are involved, they struggle to maintain information due to vanishing/exploding gradients.

As RNNs evolved as language processors, we often had cases where we needed a model that could handle even longer sequences, as text data is usually thousands of characters.

In order to accommodate the upcoming longer sequence length, researchers decided to design new architectures such as the Long Short-Term Memory units (LSTMs) or the Gated Recurrent Units (GRUs).

### LSTMs and GRUs

The GRU and LSTM handle the vanishing gradient problem by incorporating specific gates that have a well-defined purpose.

An LSTM has three or four gates depending on the source you consult. For most cases, we mainly define three gates: the input gate (it), the forget gate (ft), and the output gate (ot).

However, there is also a fourth component that is called either the input node or the candidate cell state (~Ct).

The gates are described below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/957e7621-1fd6-4b25-98ae-53f44708fe9d/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x___4_.png)

All three gates have about the same equation structure, while Wi, Wf, and Wo are weight matrices specific to each gate, bi, bf, and bo are the biases specific to each gate, and σ is the sigmoid function.

For an LSTM, the cell or internal state (Ct) is the main equation tying the different gates together for its enhanced memory capability. Its equation is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c595dd01-f0bb-4105-87e2-1ef44fe1637d/image.png)

Where ft represents the forget gate, it represents the input gate, Ct-1 is the previous internal state, and ~Ct is the candidate cell state. As a quick reminder, the circle with the dot in the middle (⊙) represents the Hadamard or element-wise product.

The candidate cell state ~Ct is a temporary value (which uses the input) that is potentially added to the current cell state. It’s calculated using the input and the previous hidden state and uses the tanh activation function.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0438c4c2-5019-4464-b131-71578bf8c282/image.png)

Here, Wc are the cell state weights, bc is the cell state bias, and the brackets represent the concatenation of the previous hidden state ht-1 and the input xt.

The hidden state ht is calculated by passing the cell state or internal state Ct through the tanh activation function, then multiplying it element-wise with the output gate (ot).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/567f515b-a10b-4abf-bc17-deb44bfa9092/image.png)

Note that for LSTMs, the hidden state is also used as the output (which is how the output gate controls the output).

Here is how an LSTM looks in a diagram format:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d4eae919-56c7-411f-b9bc-ea6ba8a4b9c9/Copy_of_Copy_of_Copy_of_Add_a_heading__4_.png)

Now what about the GRU? The GRU offered a simpler and more efficient version of the LSTM cell that often achieves comparable performance but with the advantage of being faster to compute.

GRUs only have two gates: the update gate (zt) and the reset gate (rt).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8963233e-e423-49d3-b4dc-a58451051163/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png)

Similar to the LSTM, both gates are about the same as equations. Wz and Wr are the update and reset weight matrices respectively; bz and br are the update and reset biases respectively; and σ is the sigmoid function.

The GRU’s hidden state (ht) is calculated a bit differently from the LSTM’s. It is based also on a candidate state but does not require an additional cell or internal state.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/89eb45f8-ea0f-4c2a-a27d-8fd80a4a2b2a/image.png)

Above, zt is the update gate, and h̃t represents the candidate hidden state. The candidate state is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/402aa725-acea-4d23-b768-ea841d8fba2a/image.png)

Here, Wh represents the hidden state weight matrix, rt is the reset gate, bh is the hidden bias, and the brackets denote a concatenation. We still use the tanh activation function.

Like the LSTM, the hidden state is used as our output. The entire GRU is shown in diagram form below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18fb0257-e735-435c-af5e-34d8a15dfcdb/Copy_of_Copy_of_Add_a_heading__25_.png)

Want to learn more about RNN architectures such as the GRU or LSTM? You can find our issue explaining these [here](https://www.aibutsimple.com/p/recurrent-neural-networks-revisited?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-simple-guide-to-rnns).

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-simple-guide-to-rnns):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-simple-guide-to-rnns)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-simple-guide-to-rnns)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team