# Quantization for LLMs, Simply Explained

#61 Jul 28, 2025 Inference

https://www.aibutsimple.com/p/quantization-for-llms-simply-explained

Model quantization is a technique used to reduce the size of large neural networks—especially Large Language Models (LLMs)—by modifying the precision of their weights.

Large Language Models’ size is determined by the number of parameters they have. Most LLMs live up to their name—they are in fact large. You might’ve heard that GPT-3 has 175 billion parameters.

To account for the sheer number of parameters, LLMs generally need many GPUs with huge amounts of VRAM. Additionally, inference at scale also benefits from more of these GPUs.

Parameters are simply numbers stored in memory—each parameter has a distinct storage size. This size is determined by the number’s precision.

There are different levels of numerical precision, such as 32-bit floats, 16-bit floats, or 8-bit floats, with 8-bit floats taking up the least space in memory.

The idea lies here: if we decrease the precision and thus the size of our numbers storing the parameters, we can decrease the size of the model overall and therefore need less compute to store and run it.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/791907da-2029-4f22-b966-e9ff06c376d9/Quantization.png)

On the surface level, this sounds great! However, the issue is that if we naively decrease the precision, the accuracy and performance will drop. Consider this: if our weights are originally decimals, how can we expect to achieve the same accuracy if we start storing whole numbers?

Since we still want to maintain an acceptable level of accuracy with our model, we use clever quantization techniques to achieve it.

---

### [Vibe Code With Fellow Creators On YouWare](https://www.youware.com/?utm_source=omane-media&utm_medium=newsletter&utm_campaign=Influencer&utm_content=aibutsimple-subscribe)

YouWare is a world-first vibe coding community with 100K+ creators, 300K+ projects, and AI agents, where coding, sharing, gaining traffic, feedback, and connecting happen all in one place. YouWare features powerful built-in AI tools to visualize creative projects, from ideation and design to coding, debugging, and deployment with a shareable URL.

[Start Vibe Coding With YouWare Today!](https://www.youware.com/?utm_source=omane-media&utm_medium=newsletter&utm_campaign=Influencer&utm_content=aibutsimple-subscribe)

---

### What is Quantization?

The term quantization itself refers to the process of mapping continuous infinite values to a smaller set of discrete finite values.

In the context of LLMs, it refers to the process of converting parameters from higher-precision data types to lower-precision ones.

We know that LLMs are represented in the memory of the GPU or RAM as tensors—multidimensional arrays of numbers. Within the tensors themselves, we may use different types of numbers: Float32 (often called full precision), Float16 (half precision) or even Integers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c4a84ba8-de1f-49a2-bb73-ba1d4fb6e10b/Quantization_with_Tensors.png)

The data type you choose will impact the number of "digits" that need to be used in memory and, of course, how much memory the number takes up. This is what we refer to as precision.

Essentially, the more digits, the more precise. The fewer digits, the less precise.

### How are Numbers Represented on Computers?

On computers, numerical values are typically represented as floating-point numbers, numbers with decimal points that can be both positive and negative.

We call these floats in computer science. Floats can handle very large and very small numbers, which is useful for scientific applications.

These numbers are made up of binary digits (0 or 1), which we refer to as “bits.” Bits are the fundamental, indivisible units of computer memory (other units include bytes, megabits, gigabytes, etc.).

In floats, the IEEE-754 standard says bits can be used for 3 different functions: the sign, the exponent, and the fraction/mantissa.

We use the different types of bits to calculate the actual value given a certain set of bit values. The float 16-bit, or FP16, allocates 1 bit to the sign, 5 bits to the exponent, and 10 bits to the fraction.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/84418001-6087-464d-aeb9-8790d13841d3/Numbers_represented_on_PC__2_.png)

The more bits we use to represent a value, the more precise it generally is. Also, given more bits, a larger range of values can be represented. For instance, float 32-bit, or FP32, can represent a larger range of values than FP16 due to more bits allocated to the exponent.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ac0d5b3-3bfd-4fc6-8663-f1f8c1617710/FP32_vs_FP16__1080_x_650_px_.png)

There are 8 bits in a byte of memory, so using some math, we can work out the approximate amount of memory (VRAM) a model might take up. In practice, many more things other than parameters take up memory, so this is just a simplification.

We take the number of bits of the data type, multiply it by the amount of parameters of the model, then divide the total by 8, since there are 8 bits in a byte.

Now let’s assume that we have a model with 175 billion parameters, the same amount GPT-3 had. Since most models are natively represented with 32-bit floats, which would require 700GB of memory just to load the model.

From this, it makes a lot of sense to attempt to minimize the number of bits representing the parameters of your model!

### Importance Of Quantization

This is where quantization comes in. We quantize full-precision or half-precision floating-point numbers (FP32, FP16) to lower-accuracy (lower-bit) integers.

Quantization significantly decreases the model's size by reducing the number of bits required for each model weight. Quantization aims to reduce the number of bits needed to represent the original parameters while preserving as much precision as possible.

This allows for models to run on cheaper hardware and with higher speed. However, as previously mentioned, by reducing the precision of the weights, the overall quality of the LLM can also suffer some impact.

- Studies show that this impact varies depending on the techniques used and that larger models suffer less from change in precision.

### Data Types (FP32, FP16, etc.)

First, let’s look at 4 common data types and analyze their range, decimal count, and how bits are allocated to the sign, exponent, and fraction.

Our data types are:

- Floating-Point 32-bit (FP32)
- Floating-Point 16-bit (FP16)
- Brain Floating Point Format (BF16)
- Integer 8-bit (INT8)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6a1bcabd-d155-4ec5-acd7-689c1aef8d53/FP32_vs_FP16_vs_BF16_vs_INT8.png)

FP32 has a high range and can represent a large amount of decimals accurately. Going from FP32 to FP16, notice how the range of values is a lot smaller: we went from being able to represent values from -3.4×1038 to +3.4×1038 to only being able to represent values from -65504 to +65504.

To get a similar range of values as the original FP32, “Brain Floating Point Format,” or BF16, was introduced ([by Google Brain](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained)) as a less precise version of FP32.

- FP32 can represent up to about 7 decimal points with full accuracy, while BF16 can only represent 2-3 decimal points.

BF16 uses the same amount of bits as FP16 but can take a wider range of values and is often used in deep learning applications.

To reduce the number of bits further from 16, we have to start using integers. Going from FP32 to INT8 shrinks the range to a measly 255 integers total—the ability for decimals is also removed.

Although the range is decreased, going from FP32 to INT8 results in a 4 times reduction in bits!

- Integers are also useful since calculations with them might be faster than floating-point calculations depending on the hardware, and computations are generally faster using fewer bits.

But to actually go from FP32 to INT8, we need to map the initial FP32 representations into INT8 representations.

Common mapping methods include symmetric and asymmetric quantization. These are forms of linear mapping. Both methods of mapping are applicable in many types of quantization, which we will discuss later on.

### Mapping Methods

#### Symmetric Quantization

In symmetric quantization, the original floating-point values are quantized to a symmetrical range centered around zero. Zero in the floating-point space is exactly zero in the quantized space.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b0bb36e0-a14a-403f-8687-fdf9b67208a3/Symmetric__1_.png)

Note: The symmetric INT range is from [-127, 127].

We’ll use a type of symmetric quantization called absolute maximum (absmax) quantization for this example.

In absmax, given a matrix of values, we take the maximum absolute value (α), whether negative or positive, then set it as the maximum and minimum values. The range then becomes between -α and +α.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b8bfc614-da66-4b80-b2cd-a1bea2394dba/Symmetric_Example__3_.png)

The formula for symmetric mapping using absmax is very straightforward. We first calculate a scale factor (s) using the number of bits that we want to quantize to (b) and our previously obtained maximum absolute value (α).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/73e8ae9b-5079-4bf3-ac6d-80342e66d98d/Calc_Symmetric__2_.png)

Going from FP32 to INT8, we set b to be 8. We then use the scale factor to quantize the input xfloat to xquant.

Let’s do an actual numerical example using the same 5 values used in our previous example.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1d253bab-1439-4927-af5a-3d2ef7a8d023/Copy_of_Calc_Symmetric.png)

To retrieve the original FP32 values, we simply multiply the previously calculated scaling factor (s) with the quantized values (xquant) to dequantize them.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7cb92f57-1156-4c87-8925-c90455b2b3d9/Copy_of_Symmetric_Example__1080_x_800_px___1_.png)

When you dequantize the values to return to FP32, numbers lose some precision and are not exactly the same as the original inputs. There is a slight difference, which we refer to as the quantization error.

We can calculate the quantization error using the difference between the original and dequantized values. Generally, the higher the reduction in bits, the more quantization error we tend to have.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained)

---

#### Asymmetric Quantization

Asymmetric quantization does not result in a symmetric range around zero. Instead, it maps the minimum (β) and maximum (α) values from the floating-point value range to the minimum and maximum values of the quantized range.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8a089228-effb-4693-bc22-26a9fd153a87/Asymmetric_Example__1_.png)

Above, you can see how the 0 has shifted positions—hence asymmetric quantization. The min/max values have different distances to 0 since now α and β are different.

In asymmetric quantization, we have to compute an additional variable called the “zero-point.” The zero-point represents the integer value where 0 lies on the original floating-point range.

The asymmetric quantization formulas are a little bit different than the symmetric. The scaling factor (s) is calculated using the difference of the maximum and minimum values.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/97d7f9ee-e9c0-47b7-971c-cdf300d269d3/Calc_Asymmetric__1080_x_660_px_.png)

The zero-point (z) uses the scaling factor (s) in its calculation. Because we have this zero-point, our quantized values must also take it into account.

To dequantize an asymmetric quantization, we need to use both the scale factor (s) and zero-point (z).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/05e25688-4077-42a9-a01d-17cd7486dba3/Copy_of_Copy_of_Calc_Symmetric__1080_x_450_px_.png)

Both asymmetric and symmetric quantization can be applied to different types of quantization algorithms. Let’s dive into some common LLM quantization algorithms to quantize weights and activations of a network.

### The Two Types of LLM Quantization

There are two main types of quantization for LLMs: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

In PTQ, we perform quantization after training. PTQ uses either symmetric or asymmetric quantization to convert the weights of an already trained model to a lower precision without any retraining.

For PTQ, we typically use a calibration dataset (a small representative dataset) to observe the range of values of the weights and activations. Using this range, we calculate statistics such as the scale factor or zero-point, allowing us to quantize the values using our formulas.

PTQ is straightforward and easy to implement, but it can lead to a noticeable drop in the model's performance due to loss of precision.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/62c70068-8c2c-4722-b08f-ffde142a7e98/PTQ_vs_QAT__1080_x_900_px___1_.png)

In QAT, we perform quantization during training/fine-tuning. QAT integrates the weight conversion process during the training stage—it attempts to learn the quantization parameters during training.

This often results in better accuracy, since it forces the model to adjust its weights to be resilient to the information loss from quantization. However, QAT is more computationally demanding, as it requires a full retraining or fine-tuning cycle.

### Quantization Example With LLama

Now that we understand what quantization is, how it works, and the different ways of compressing numbers, let’s take a look at how much memory quantization can actually save.

Let’s assume we want to store a LLama2-70B model, which takes up 138GB of memory on FP16. To store this model, you would need multiple Nvidia A100 GPUs, since an A100 only has 80GB of VRAM in its top-of-the-line version.

Not only are the GPUs expensive, but distributing models over multiple GPUs also means paying for overhead infrastructure.

A quantized version (INT4) of the LLama2-70B requires only around 40GB of memory, fitting into a single A100 GPU. Using the quantized model would also result in faster computations, speeding up overall execution.

Here is a table showing different sizes of LLama models and their respective quantized sizes:

| Model | Original Size (FP16) | Quantized Size (INT4) |
| --- | --- | --- |
| Llama2-7B | 13.5 GB | 3.9 GB |
| Llama2-13B | 26.1 GB | 7.3 GB |
| Llama2-70B | 138 GB | 40.7 GB |

---

Visuals in this issue inspired by [Exploring Language Models, a newsletter by Maarten Grootendorst](https://newsletter.maartengrootendorst.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained).

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=quantization-for-llms-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team