# Recurrent Neural Networks, Simplified #8 Jul 1, 2024 Architecture https://www.aibutsimple.com/p/recurrent-neural-networks-simplified A recurrent neural network (RNN) is a deep learning model that is trained to take a sequential data input and transform it into a specific sequential data output. - Sequential data is any type of data where the order of the data is important. These can be words, sentences, or time-series data. The key feature of RNNs is their ability to maintain a hidden state that captures information from previous time steps, effectively creating a form of memory. They take information from prior inputs to influence the current input and output. This hidden state is updated at each time step as the network processes new input data, enabling the network to learn from both the current input and the historical context. Some common applications of the RNN include language translation, natural language processing (NLP), image captioning, and speech recognition. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a3a808de-dc96-42b4-9010-f9739ed3fbf1/image.png) Above, the hidden state is visualized by the loops So why are RNNs used specifically for sequential data? Since they have this hidden state and a type of memory, it helps them process sequences of data where the order and context matter (text, time series data, etc.) better than other neural network architectures. Another distinguishing characteristic of recurrent networks is that they share weights across each layer of the network. In a standard feedforward neural network, each neuron in each layer has its own unique weight matrix for its connections to the previous layer. This means that every node independently learns its weights based on the inputs it receives. In contrast, in an RNN, each node in a layer uses the same weight matrices for its connections across different time steps. This means that as the RNN processes each element in a sequence, it applies the same set of weights repeatedly. - This weight sharing allows the network to capture patterns within the sequence efficiently, which is another reason to use this network for sequential data. The weights are still adjusted during training through backpropagation and gradient descent, with slight modifications. ### Architecture --- Let’s investigate a very simple RNN architecture. This RNN will have an input layer, a hidden layer, and finally an output layer. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e85fdaa1-9059-4fd2-b122-a62059a4bcbd/Copy_of_Add_a_heading__8_.png) For the input layer, a vector xt​ is taken as the input to the network at time step t. - The vector depends on your use case. For instance, in natural language processing, each data point could be a word represented by a vector. In the context of an RNN, a time step (t) refers to a single point in a sequence. For example, if you are processing a sentence word by word, each word represents a time step. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8a2e50d5-54f1-425c-a058-913ea2ae29c8/Copy_of_Add_a_heading__5_.png) The hidden layer maintains a hidden state ht​ that is updated at each time step. This hidden state captures information from previous time steps. - The hidden state ht​ at time step t is computed based on the current input xt​ and the previous hidden state ht-1​ - This ht-1 is how the hidden state gains its “memory” - ht​ = f(Wxh*xt + Whh*ht-1​), where Wxh is the weight matrix of the input to hidden state, and Whh is the weight matrix of the hidden to hidden state. The function f is taken to be a non-linear transformation such as tanh or ReLU. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd3459ed-961f-468e-9d89-5e1fc838a9a1/Copy_of_Add_a_heading__10_.png) These matrices are multiplied with either x(t) or h(t-1). There can also be a bias b, which is not displayed For the weights, the RNN has input to hidden connections parameterized by a weight matrix Wxh, hidden to hidden recurrent connections parameterized by a weight matrix Whh, and hidden to output connections parameterized by a weight matrix Who and all these weights are shared across time. - These weights are initialized at random and serve as an important aspect of the inner node architecture. These input to hidden, hidden to hidden, etc, all simply describe the connection of the neuron: input to hidden describes the connection into the neuron, hidden to hidden describes the loop back into itself, and hidden to output is the outgoing connection. - Visually, the loop connection at each neuron is represented by the hidden to hidden weight matrix multiplied by ht-1, where the hidden state from the previous time step is fed back into the neuron to influence the current hidden state. The output, ot​ is calculated from the hidden to output weight matrix multiplied by ht added to some bias value c. ot is often passed through more activation functions if the network has more layers. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3d8c0928-f8b9-4f95-b666-5b5599b2eacd/Copy_of_Add_a_heading__7_.png) Keep in mind that this matrix is multiplied by h(t) During the entire forward pass, the input sequence is processed one time step at a time, and can be described by the image below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/34b7b29a-e81d-4165-a2d7-273f652833a8/image.png) at is the result of the computations involving the hidden to hidden and input to hidden weight matrices, and ht involves passing it through an activation function. Then, we use a hidden to output weight matrix Wxo multiplied by ht , added to a bias c, to compute the output. - This can be propagated forward until it reaches the end of the network, where the prediction (ŷ) can be a probability distribution of all possible output classes (from the softmax function). The initial hidden state h0 is typically initialized to a zero vector. Then, t will start from 1 to start the computation process (use h0 and x1). ### Backpropagation Through Time (BPTT) --- Training an RNN involves adjusting the network's weights to minimize a loss function. This is done using a variant of backpropagation called Backpropagation Through Time (BPTT): The backward pass begins by unrolling the RNN in time, treating it as a multi-layer feedforward network where each layer corresponds to a time step. - This means that the loop is unrolled into many other layers, each with a corresponding time step: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a3379135-228a-4387-ac19-63f25fdb8c39/image.png) L is any loss function, and U, V, and W are the input to hidden, hidden to hidden, and hidden to output state matrices Gradients of the loss with respect to each weight are computed by backpropagating the error through the unrolled network. Then, the weights are updated using gradient descent or a related optimization algorithm. If you want to go deeper in the math of BPTT, feel free to read this [resource](https://jramapuram.github.io/ramblings/rnn-backrpop/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=recurrent-neural-networks-simplified) ### Problems with RNNs --- Despite their power, standard RNNs can struggle with long-term context due to issues like vanishing/exploding gradients. - During training, the gradients used to update the weights can become very small, causing the learning process to stall. - The gradients can also become very large, causing the model weights to grow uncontrollably and making the training process unstable. - To learn more about vanishing and exploding gradients, feel free to check out [this](https://www.aibutsimple.com/p/deep-learning-overfitting-underfitting-vanishing-gradient?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=recurrent-neural-networks-simplified) previous issue. They can also be computationally intensive and slow to train, especially for long sequences. To address this, more advanced variants such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) were developed. These architectures include gating mechanisms that help the network learn to retain or forget information as needed, making them more effective for capturing long-term context in the data. --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=recurrent-neural-networks-simplified) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team