Recurrent Neural Networks, Revisited

One of the big problems with the original RNN design is when long sequences are involved, as they struggle to maintain information due to vanishing/exploding gradients.

As RNNs shifted more into language processing, we often had cases where we needed a model that could handle longer sequences, as text data is usually thousands of characters.

• Due to this inability to maintain information over long sequences, RNNs tend to “forget” information from earlier parts of a sequence, even if it is crucial.

If you don’t know much about RNNs and want to learn more about them before continuing (time steps, hidden state, etc.), you can check out our issue about them here.

In order to accommodate the upcoming longer sequence length, researchers decided to design a new and complex architecture: the LSTM.