Transformers and the Attention Mechanism

The earliest years of the deep learning boom were driven primarily by the Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) architectures.

Nowadays, a different type of model is being used widely and has become very popular, with examples like ChatGPT, BERT, and Gemini. This model has completely taken the deep learning world by storm, and it is called a transformer.

A transformer model is a type of neural network architecture that has become the foundation for many modern natural language processing (NLP) tasks, such as translation, text generation, and chatbots.

The core idea behind the transformer model is the attention mechanism, an innovation that was originally created for encoder–decoder RNNs for machine translation.

Unlike earlier models like RNNs or LSTMs (Long Short-Term Memory networks), transformers do not process data sequentially.

Instead, they rely on an attention mechanism to process all words (or tokens) in a sentence at the same time, which greatly increases the performance on natural language processing (NLP) tasks.

Here’s a quick timeline on the attention mechanism and transformers:

• The attention mechanism was first used in 2014 in computer vision.

• In 2015, we see use of the attention mechanism in machine translation, a standard NLP task.

• In 2017, the attention mechanism was used in the first transformer network in the famous paper “Attention is all you need.”

• Transformers have since surpassed the performance of RNNs greatly for NLP tasks.

So how did transformers end up dethroning the RNN for NLP tasks? Well, they solved a load of problems the RNN couldn’t handle.

Problem: RNNs encounter issues with long-range data. RNNs do not work well with long text documents.

• Solution: Transformer networks use many attention blocks, making long-range dependencies have the same likelihood of being taken into account as any other short-range dependencies. So instead of sequence-to-sequence models like RNNs, who only feed the final hidden state, we feed transformers every token to be able to develop a larger picture of the data.

Problem: RNNs suffer from the vanishing and exploding gradient problem.

• Solution: Transformers rarely have gradient vanishing or explosion. They use skip/residual connections that allow it to access the input (read this issue for more on residual connections).