Issue #8 Architecture Edwin Dong
Recurrent Neural Networks, Simplified
A recurrent neural network (RNN) is a deep learning model that is trained to take a sequential data input and transform it into a specific sequential data output.
Sequential data is any type of data where the order of the data is important. These can be words, sentences, or time-series data.
The key feature of RNNs is their ability to maintain a hidden state that captures information from previous time steps, effectively creating a form of memory.
They take information from prior inputs to influence the current input and output.
This hidden state is updated at each time step as the network processes new input data, enabling the network to learn from both the current input and the historical context.
Some common applications of the RNN include language translation, natural language processing (NLP), image captioning, and speech recognition.
Above, the hidden state is visualized by the loops
So why are RNNs used specifically for sequential data? Since they have this hidden state and a type of memory, it helps them process sequences of data where the order and context matter (text, time series data, etc.) better than other neural network architectures.
Another distinguishing characteristic of recurrent networks is that they share weights across each layer of the network.
In a standard feedforward neural network, each neuron in each layer has its own unique weight matrix for its connections to the previous layer. This means that every node independently learns its weights based on the inputs it receives.
In contrast, in an RNN, each node in a layer uses the same weight matrices for its connections across different time steps.
This means that as the RNN processes each element in a sequence, it applies the same set of weights repeatedly.
This weight sharing allows the network to capture patterns within the sequence efficiently, which is another reason to use this network for sequential data.
The weights are still adjusted during training through backpropagation and gradient descent, with slight modifications.
Architecture
Let’s investigate a very simple RNN architecture. This RNN will have an input layer, a hidden layer, and finally an output layer.
For the input layer, a vector xt is taken as the input to the network at time step t.
The vector depends on your use case. For instance, in natural language processing, each data point could be a word represented by a vector.
