# Transformers and Self-Attention, Mathematically Explained

#38 Feb 17, 2025 Architecture

https://www.aibutsimple.com/p/transformers-and-self-attention-mathematically-explained

This week’s issue is a continuation of last week’s topic on transformers, exploring the inner workings and the math computations within the most popular model for NLP tasks. The first part can be found below:

- [Transformers, Mathematically Explained (Intermediate)](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=transformers-and-self-attention-mathematically-explained)

Here’s a quick refresher on transformers: A transformer model is a type of neural network that is popularly used for many modern natural language processing (NLP) tasks for its state-of-the-art performance.

- Some examples of transformers used in the world are chatbots (such as ChatGPT), translation tools, and text generation.

The key behind why the transformer works so well for language is the self-attention mechanism—originally created for RNNs and machine translation.

- The attention mechanism allows them to weigh the importance of different tokens (words or symbols) in an input sequence when making predictions.
- This weighting lets the model understand the context of individual words better—capturing relationships between words in longer texts.

Last week, we went over a simple worked example using a small vocabulary and a 3-word sentence: “the cat is”. In that issue, we covered the computations within the transformer encoder.

This week, we’ll be discussing the transformer decoder, covering all the math and computations.

---

Again, before continuing, it is recommended to understand how transformers work conceptually, as this issue is long and includes a worked example with math. We have an issue introducing transformers below:

- [Introduction to Transformers and the Attention Mechanism (Intermediate)](https://www.aibutsimple.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=transformers-and-self-attention-mathematically-explained)

---

### Transformer Decoder

Take a look at our transformer architecture below. We already covered the encoder (left side); now we’re going over the decoder (right side):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/421d5d5c-1cad-4db3-8ebd-c3ca5ab3777f/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___1_.png)

In our encoder, we only performed unique computations, from encoding our dataset to passing our matrix through a feedforward network.

In the decoder, the good thing is that many of the computations are the same, which we’ve already covered in the encoder.

To avoid repetitive steps, we won’t be repeating them in full but instead focusing on some key differences between the encoder and decoder.

- We’ll focus on the calculations of the input and output of the decoder, as well as some specific blocks in the center of the decoder.

#### Decoder Input Streams

The decoder takes two main streams of data as input.

The first is from the encoder, where the output matrix of the last “add and norm” layer is multiplied with different query (Q) and key (K) weights to act as the key and query matrices for the second multi-head attention layer in the decoder:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c7e5f0ae-4f10-4e4d-b8d7-b540d8a62fb4/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___3_.png)

As with the other multi-head attention blocks, in addition to the key (K) and query (Q) matrices, a value (V) matrix is also used.

This value (V) matrix is produced after taking entries from the previous “add and norm” step in the decoder and multiplying by another weight matrix.

The second input to the decoder is predicted text. Essentially, the decoder takes some text that we want the model to predict or text that it had already predicted in previous timesteps. In our case, the text is: “sleeping comfortably”.

To refresh your memory from the first issue, our dataset contains this full sentence: “The cat is sleeping comfortably.”

For the encoder input sentence, we used the first three words: “the cat is”. Now, for our decoder input, we’ll use the last two words: “sleeping comfortably”.

This approach effectively labels the dataset, giving the transformer a supervised learning process.

The text that we feed the decoder needs to be wrapped with two tokens, a “<start>” token and an “<end>” token. This lets the transformer know where to start generating tokens and when to stop.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4bca46a2-c8ca-429a-9da1-cc290eceb56a/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_300_px___720_x_400_px___2_.png)

The decoder works autoregressively, meaning it predicts one word at a time based on words that have been generated so far. To make this work, it takes one input token at a time. In our case, “<start>” will be served as the singular input token, and “sleeping” is what we hope it will predict.

We embed this token the same way we did for the encoder, filling a matrix with random values between 0 and 1. These values will be nudged gradually during training.

After the embedding process, we’ll use the same positional encoding we used for the encoder to embed position information into our word embedding.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6b2e58bb-5c30-46bc-986a-0a7edc07262d/csaiochaoisc__3_.png)

After the positional encoding, we compute the rest of the blocks of the decoder the same way we computed earlier in the encoder portion. Below, the greyed-out portions are parts we have already covered in the encoder.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/555aa4f7-b9b4-414b-ba2b-1d5cc5d7f5db/Copy_of_Copy_of_Copy_of_Copy_of_positional_emb.png)

---

#### Encoder Recap

Here’s a quick recap of the calculations we’ve done so far that are repeated in the decoder:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3fa1030d-faa5-49e5-9f36-bae0499f2fd8/Copy_of_positional_emb__2_.png)

We computed the self-attention mechanism by using query, key, and value matrices and multiplying according to the diagram.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f1b36dea-125b-41ef-b39b-30542590a096/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px_.png)

The final attention output is obtained by multiplying the resultant of the softmax matrix and the value matrix by some random linear weights. This is what comes out of the multi-head attention block.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f0fd907b-36b1-42c5-a0c5-f2c7acf002c4/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___3_.png)

Reminder that multi-head attention works about the same as single-head, where multiple single heads are concatenated together before multiplying by the linear weights. Both versions need to be linearly transformed.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/36ae7408-73a2-461c-8c2c-4001089ce876/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___1_.png)

We then also calculated a simple example of the feedforward section. For demonstration purposes, we used a very simple feedforward network with one layer and one activation (ReLU).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4516a25b-9495-447b-af5f-c37a2ee34ced/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px_.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c65aff0e-0d5a-4882-9aa0-061551a9a02b/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px_.png)

The “add and normalize” block simply required adding the input streams of matrices and normalizing them:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8307206e-6e53-45c8-b638-4e29298f0734/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px_.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/77fcc348-9a10-4d41-b9d7-5fd81b11f69c/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px_.png)

---

However, in the decoder, there’s a new block we haven’t seen before: the masked multi-head attention block.

#### Masked Multi-Head Attention

In a transformer, the masked multi-head attention is an alternative attention block that the model uses to “focus in” on different parts of a sentence.

The masked multi-head attention implements what we call causal masking, which “masks” or prevents the model from seeing future tokens when making predictions, creating a “left to right” or sequential learning process.

The masking is crucial since it prevents the model from “cheating” by looking at words that come later in the sentence—it doesn’t need to predict anything if it already knows what comes next. This is useful for text generation or translation, where the model must predict the next word without knowing what comes next.

The process starts by multiplying the input matrix with distinct weight matrices to obtain our query (Q), key (K), and value (V) matrices.

Keep in mind that each head will have different weights for the Q, K, and V matrices but will receive the same copy of the input matrix.

- Each head works on a different subspace of the original data, allowing the model to attend to more information and capture diverse relationships.

Then, for each head, the attention scores are computed by taking the dot product between the query and key vectors. Like the multi-head example in the encoder, these scores are then scaled by dividing by the square root of the key dimension (dk) to keep the values in a stable range during training.

After these computations, masking is applied to the resultant matrix to prevent the model from “seeing” future words—we want the decoder to generate tokens one by one.

But how does the masking actually work?

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b193855a-fa38-4bfd-9ac3-9a576a8157a5/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___1_.png)

In the masking process, the mask is typically a lower triangular matrix (with zeros on and below the diagonal and -∞ above) so that we set any values that have not been seen yet to -∞.

Above, notice how everything above the diagonal is blacked out—set to negative infinity. If you look at the first row, corresponding to predicting the word “<start>”, you’ll see that everything after “<start>” on the columns is masked out. The second row has the same pattern: everything after “sleeping” on the columns is masked.

So, at each time step, the model only has access to information or the individual tokens it has already been fed, using only this information to predict the next tokens in the sentence.

The mask in matrix form is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f104a93b-6f46-4144-9bee-213f1d474908/Copy_of_positional_emb__8_.png)

We specifically use the value of negative infinity so that when the softmax is applied next, these positions receive zero weight—the softmax function will set them all to 0.

After applying the causal mask, we apply softmax to our matrix to obtain the attention weights. Going with the attention diagram, we multiply the attention weights by the value (V) matrix to get the weighted sum for each head.

With multi-head attention, we concatenate the outputs from both heads and apply a linear transformation to obtain the final output.

Let’s illustrate the entire process using a simple mathematical example. In this example, we’ll use two heads of attention for our example for demonstration purposes.

Suppose we have the following input matrix, where each row represents a position in the sequence, and each column represents a token:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5b1f0e46-d711-4cb2-9db7-12df2ac1b967/Copy_of_positional_emb__7_.png)

As a reminder, the query (Q), key (K), and value (V) matrices are calculated by multiplying the same input by different weight matrices. Each head will have different values for Q, K, and V but will be sent the same input.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/286442d7-fe7e-41dd-bc29-555b2404e7b4/image.png)

From our input matrix, we obtain Q, K, and V and then we perform matrix multiplications matching the attention diagram to obtain S.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0cd7a2c7-1350-41d5-bce4-530e9be8acd9/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png)

Above, we’ll assume that we have some weights that will manipulate (Q1, K1) and (Q2, K2) to become S1 and S2 respectively.

We add the causal mask (M) to S1 and S2 and apply softmax to both, finally multiplying by the value (V) matrix.

We concatenate the resultant matrices O1 and O2, combining the outputs from two attention heads into a single matrix. This matrix will be multiplied with some linear weights to obtain the final output, which will be passed on to the next part of the transformer.

#### Predicting The Next Word

After passing the data through all blocks in the decoder, we can finally start to process the output.

The output matrix of the final “add and norm” block of the decoder must contain the same number of rows as the input matrix—here, we work with 4.

According to the decoder diagram, we take this output matrix and pass it to a linear layer. This gets us very close to our final predicted probabilities of each word! But before that, we need to flatten the matrix.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e0f22cdb-521a-4890-aab0-9447aa790304/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_positional_emb__720_x_450_px___8_.png)

This flattened layer will be passed through a linear layer to compute the logits of each unique word in our dataset.

- The term “logits” means the raw, unprocessed scores output by the final linear layer of the network before applying any activation function, such as softmax.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a765a744-0b1b-4067-9506-cb34097d7c0e/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___2_.png)

Once we obtain the logits, we can use the softmax function to normalize them and find the word that produces the highest probability.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6fa9d2a3-2ddb-4ceb-9d1b-f1c3ca771c18/Copy_of_Copy_of_positional_emb__1_.png)

So based on our calculations, the word the model will predict is “sleeping”, with a probability of 0.42.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ef390d1-be31-49e9-a506-54ed9facc155/Copy_of_Copy_of_Copy_of_positional_emb.png)

The predicted word will be used as the next input word for the decoder, and this process continues until the “<end>” token is predicted.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=transformers-and-self-attention-mathematically-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=transformers-and-self-attention-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team