Transformers and Self-Attention, Mathematically Explained

This week’s issue is a continuation of last week’s topic on transformers, exploring the inner workings and the math computations within the most popular model for NLP tasks. The first part can be found below:

• Transformers, Mathematically Explained (Intermediate)

Here’s a quick refresher on transformers: A transformer model is a type of neural network that is popularly used for many modern natural language processing (NLP) tasks for its state-of-the-art performance.

• Some examples of transformers used in the world are chatbots (such as ChatGPT), translation tools, and text generation.

The key behind why the transformer works so well for language is the self-attention mechanism—originally created for RNNs and machine translation.

• The attention mechanism allows them to weigh the importance of different tokens (words or symbols) in an input sequence when making predictions.

• This weighting lets the model understand the context of individual words better—capturing relationships between words in longer texts.

Last week, we went over a simple worked example using a small vocabulary and a 3-word sentence: “the cat is”. In that issue, we covered the computations within the transformer encoder.

This week, we’ll be discussing the transformer decoder, covering all the math and computations.

Again, before continuing, it is recommended to understand how transformers work conceptually, as this issue is long and includes a worked example with math. We have an issue introducing transformers below:

• Introduction to Transformers and the Attention Mechanism (Intermediate)