# A Deep Dive into Modern Transformers: RoPE, Activation Functions, Quantization

#86 Jan 26 Inference

https://www.aibutsimple.com/p/a-deep-dive-into-modern-transformers-rope-activation-functions-quantization

The core concept of large language models (LLMs) is very widely studied. Feed in a chain of tokens, process them using attention, then obtain a probability distribution over the next possible tokens to sample from.

Most understand how transformers fundamentally work, often through Vaswani et al.’s “[Attention is all you need](https://arxiv.org/abs/1706.03762?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)[,](https://arxiv.org/abs/1706.03762?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)” released back in 2017.

However, modern transformers used in LLMs like GPT-5, Gemini 3 Pro, and DeepSeek R1 have undergone significant updates and optimizations.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/01b3a04f-12af-4404-b9e1-5bc3f6785710/Modern_Transformer_Architectures__RoPE_.png)

In this issue, we’ll dive deep into these changes, why they happen, and how they improve performance.

Before we continue, let’s unpack some general terms we should understand:

- Perplexity, the metric used to measure how “confused” an LLM is by the text it sees. A low perplexity shows confidence, while a high perplexity shows confusion.
- Mathematically, perplexity is Euler’s number (e) raised to the exponent of cross-entropy loss. If a model has a perplexity of 10, it means that, on average, the model is guessing from 10 equally likely words.
- Length generalization, how a transformer or LLM may generalize (“perform well”) to longer sequences than it was trained on with the correct positional encoding.

This is part two of another “modern transformers” article. Although both can be read standalone, the other article provides some useful context. If you haven’t read it, you can find it here:

- [A Deep Dive into Modern Transformers: Optimizing Attention](https://www.aibutsimple.com/p/a-deep-dive-into-modern-transformers-optimizing-attention?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)

---

### Positional Encoding

The original transformer used fixed, absolute sinusoidal encodings, which were added element-wise to the input embedding vectors.

For a position pos and dimension i, the positional encoding was:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/07e07f05-7455-476f-8883-d2cb7ee745d7/image.png)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9ed660d0-4983-448f-a383-747ed8b8da4c/image.png)

For every even dimension, we use sin, and for every odd dimension, we use cos.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a977ff94-b3ba-4970-bd0c-16467215bab0/Standard_Positional_Encoding__1_.png)

The limitation here is that this positional method is "absolute." The model learns that a token is at index 5 or 100, but it struggles to naturally generalize the distance between tokens 5 and 100 versus tokens 505 and 600.

Because of this, the model often fails to extrapolate well to sequence lengths longer than the training example length.

#### Rotary Positional Embeddings (RoPE)

To address these problems, Rotary Positional Embeddings (RoPE) is used. Introduced by Su et al. in the [RoFormer](https://arxiv.org/pdf/2104.09864?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization), RoPE encodes position by rotating the embedding vector in a high-dimensional space instead of element-wise addition.

RoPE takes query (q) and key (k) vectors of dimension d, breaks each vector into d/2 pairs, and treats each pair as coordinates on a 2D plane.

Let m be the position of the query in the sequence, and let n be the position of the key. Note that these are relative positions, not absolute positions.

To encode the position index m, we rotate the query vector pairs by an angle mθ. We follow the same process for the key at position n, rotating its vector pairs by an angle nθ.

Here, m depends on the position of the token in the sequence, but θ is a constant called the base frequency.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f61f5688-13b7-4f2a-bfee-3c67e6365acb/RoPE__Rotary_Positional_Embedding__Positional_Encoding__2_.png)

Mathematically, for a 2-dimensional slice of the query or key vector, the transformation is a geometric rotation, represented by a counterclockwise rotation matrix in R2 :

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7c931a1d-32fc-4026-8706-a443460b99d4/image.png)

Here, q0 and q1 are the entries of a pair created from the query vector. We perform a matrix multiplication of the rotation matrix and the pair, resulting in our RoPE-encoded vector.

This extends to higher dimensions (queries and keys usually have 512 dimensions), using a block diagonal matrix that splits the d-dimensional space into d/2 subspaces.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/430b1fc0-6e4b-416a-a767-ab94820957c3/image.png)

This correctly “assigns” adjacent entries of the entire query/key vector as pairs.

After performing the multiplication of this matrix with the query or key vector, each pair is rotated using the 2D rotation effect as demonstrated previously.

Why is this rotation actually effective? When we compute the attention score (dot product) between a query at position m and a key at position n:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2d8f9336-9917-4354-b8ee-9bacb6b775e0/image.png)

Notice the 2nd to 3rd steps: Rm transpose multiplied by Rn is simply rotating the vector by some θ scaled by n (nθ) counterclockwise, then rotating that same vector by the same θ scaled by m (mθ).

So, the rotation can be simplified to the rotation counterclockwise of the difference between n and m, (n-m)θ.

This results in the attention score depending only on the relative distance between n and m, not the absolute positions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/85d559e1-ba74-42f9-a6f8-95868bdb6817/RoPE_Relative_Distance_vs_Absolute.png)

Shifting from absolute positions to relative positions using RoPE allows the transformer to calculate attention effectively:

- As long as the model has seen similar relative distances to previous words (e.g., 5 steps), it can produce meaningful attention values even for large absolute positions it has never seen before (e.g., position 3000).

As a result, RoPE gives newer LLMs better length generalization (performing well on longer sequences than it was trained on) and performance on long-context tasks.

It obtains much lower perplexity for higher context lengths than regular positional encoding.

### Activation Functions: No More ReLU/Sigmoid

While the original transformer uses the ReLU activation function, modern models (BERT, Llama, GPT-5, etc.) have largely replaced it with GeLU or Swish/SwiGLU variants for better performance.

Recently, SwiGLU has been widely adopted by models like PaLM and Llama, as it is slightly more computationally efficient than the popular GeLU counterpart.

SwiGLU is a variant of the Gated Linear Unit (GLU) using the Swish activation function. This replaces the standard 2-matrix (2-layer) FFN with a 3-matrix structure.

First, the Swish function is a smooth, non-monotonic curve defined as x*σ(βx):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/79c8c71c-cb26-43ce-b8b9-0ac64a042bb6/Swish_Activation.png)

Here, σ is the sigmoid function, and β is a constant hyperparameter that controls the shape of the function. When β = 1, the function is known as the SiLU.

The SwiGLU layer operates by creating two parallel projections of the input x:

1. A "gate" projection passed through Swish.
2. A "value" projection, which is simply linear.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a6b7e07f-4bca-4ce7-a86e-d76fee4ce417/SwiGLU_activation__1_.png)

These are then multiplied element-wise. This structure acts as a learnable gating mechanism:

- Through training, the model can learn what data should be passed through the value portion and which ones should be gated, leading to better performance and robustness.

One key difference is that SwiGLU requires 3 matrix multiplications (W, V, Wout) per layer instead of the standard 2 (W, Wout). This explains the extra matrix (V) added by the gating portion required in the SwiGLU block.

To keep the parameter count constant with the standard transformer FFN block, the hidden dimension size (number of nodes in the hidden layer) is usually reduced by a factor of 2/3.

Despite the complexity, SwiGLU consistently yields lower perplexity for the same compute budget.

### Quantization

As LLMs continue to grow larger, quantization has become increasingly useful to keep larger models accessible within a few enterprise-grade GPUs or even, in some cases, consumer-grade GPUs.

Instead of the regular FP32 or FP16 to INT8 or BF16, we can obtain similar performance with even more compression. New quantization algorithms now allow compression down to 4 bits with minimal performance loss.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9d0f00f1-0c42-459b-ac17-d6397152ddd6/Quantization_Modern_Transformers__1_.png)

For instance, OpenAI’s gpt-oss is using the MXFP4 microscaling quantization algorithm. MXFP4 is a block-based quantization format.

In traditional quantization (like INT8), we usually have one scaling factor for an entire channel (e.g., 4096 elements) of a weight matrix/tensor. This isn’t very precise since outliers can skew the scale, pushing smaller numbers to 0.

In MX (Microscaling) formats, you break the weights into small blocks (vectors) of 32 numbers. Each individual weight inside the block is compressed to just 4 bits, with 2 bits for the exponent, 1 mantissa bit, and 1 sign bit. This format is referred to as E2M1.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7a067d15-3405-48a1-9a5e-7b38419a3d8c/MXFP4_Quantization__1_.png)

In each block, all 32 elements share one 8-bit scaling factor, with all 8 bits used for the exponent.

- Because scaling happens locally every 32 numbers, a large outlier only affects its immediate neighbors, maintaining the accuracy of the rest of the model.

Due to this local scaling, MXFP4 is found to be more accurate than standard 4-bit integer quantization (INT4), which has larger element windows. The floating-point representation also handles small and large numbers more effectively than integers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a890f59e-d277-46c2-b088-98bc155daaff/MXFP4_Quantization_Local_Performance__1_.png)

In gpt-oss, MXFP4 is used in the Mixture-of-Expert (MoE) blocks, allowing the massive LLMs to fit on much less hardware.

- The 20B parameter model (16GB) can fit on a single RTX 4080 GPU, while the 120B (80GB) parameter model can be held by one 80GB VRAM GPU (A100, H100, or newer).

While the large model may not exactly fit on “consumer hardware,” it is a cost-effective solution for small businesses or research groups, as opposed to multi-H100 machines or costly API calls.

- Without MXFP4 optimization, the 20B model in BF16/FP16 would consume over 40GB, and the larger 120B model would consume over 240GB.

Other modern quantization methods include AWQ, GPTQ, and GGUF, which are implemented in models like Llama 3, Mistral 7B, Falcon 180B, and Qwen. If you want to learn more about quantization, check out our in-depth issue on quantization [here](https://www.aibutsimple.com/p/quantization-for-llms-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization).

### Normalization

The original transformer (Vaswani et al., 2017) and BERT placed the normalization layers after the attention block and feedforward block, known as Post-LN (Post Layer Norm).

In Post-LN, normalization happens after the residual addition, forcing the output of every layer to have a mean of 0 and a variance of 1.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f70298f2-b8e9-45e5-87c0-68e5d41165ca/Post_Norm__2_.png)

However, theoretical analysis shows that in Post-LN, the gradient norm is dependent on the parameters of all layers. As the network gets deeper, gradients near the input fluctuate a lot, making training unstable.

- By placing normalization inside the residual block, the skip connection remains untouched.

Mathematically, this creates a direct path for gradients to flow from the loss function to the input layers without being “squished” by normalization, significantly stabilizing training.

GPT-3 and other modern LLMs that came after it implemented this by placing the normalization layers before the attention and feedforward blocks, known as Pre-LN.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/24f44ae1-b521-4bc6-8f5b-c061f9e98db9/Pre-LN_vs_Post-LN__1_.png)

While Pre-LN fixed the gradient flow issues, researchers realized the normalization calculation itself was inefficient. This led to the adoption of Root Mean Square Normalization (RMSNorm) in models like Llama and Gopher.

Standard LayerNorm ([Ba et al., 2016](https://arxiv.org/pdf/1607.06450?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)) enforces a mean of 0 (re-centering) and a variance of 1 (re-scaling).

RMSNorm simplifies the process by removing mean centering. It normalizes purely based on the Root Mean Square magnitude (RMS). The RMS is defined as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ad09afb-3282-409b-a4c5-8da9fe034f2c/image.png)

The normalization in RMSnorm is achieved by dividing the data by the RMS magnitude and multiplying it by some constant γ.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7df7a1c6-f952-4ae7-ab7d-7cbfc33d8e80/image.png)

RMSNorm has certain benefits over regular layernorm.

First, RMSnorm preserves the property that if you scale the input by a constant c, the output remains unchanged, since the RMS also scales by c, cancelling it out.

- We call this scaling invariance, which helps keep the model’s learning stable.

Also, by removing the calculation of the mean and a bias term, RMSNorm reduces the amount of computations needed. In LLMs, this can speed up execution time by 10-20%, and in certain cases, even more.

An increasing number of modern LLMs now use RMSNorm, including DeepSeek V3, Llama 4, and gpt-oss.

Another change to normalization seen in many modern LLMs is the removal of dropout.

Dropout is a common technique in traditional deep learning used to prevent overfitting, initially introduced to transformers in the original 2017 paper and adopted in early LLMs like GPT-2.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9ad813e6-5282-428d-b966-6c9af9d5f266/Dropout_Modern_LLMs.png)

These days, training dataset sizes reach trillions of tokens, reducing the risk of overfitting. Since overfitting is no longer the primary risk, dropout is now removed to save compute and to improve training stability.

### Recap

So far, we’ve explored major changes to the transformer architecture used in modern LLMs.

RoPE is replacing absolute sinusoidal positional encodings, improving performance using relative rotation calculations rather than absolute addition.

SwiGLU and GLU variants are now dominating modern LLMs, replacing the old ReLU activation function, yielding consistently lower perplexity scores for the same compute budget.

Block-based quantization methods such as MXFP4 have increased in popularity, handling a large range of values with decent accuracy, decreasing the size of LLM weights while maintaining as much performance as possible.

Researchers have also switched to Pre-LN normalization, opting for a scaling-invariant, computationally efficient normalization method such as RMSnorm. Dropout has now been phased out of most LLMs.

Together, these optimizations have enabled smaller, faster, and significantly more efficient modern transformers, supporting the high-performance models widely used today.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=a-deep-dive-into-modern-transformers-rope-activation-functions-quantization)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team
