Issue #61 Inference Edwin Dong, Ebrahim Jahangiri
Quantization for LLMs, Simply Explained
Free editionPRO edition
This week’s PRO issue is accompanied by a Colab notebook, going over a basic implementation of using a quantized LLM. The code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Feel free to play around with the code and learn more about how GPR works! You can find the notebook here.
Model quantization is a technique used to reduce the size of large neural networks—especially Large Language Models (LLMs)—by modifying the precision of their weights.
Large Language Models’ size is determined by the number of parameters they have. Most LLMs live up to their name—they are in fact large. You might’ve heard that GPT-3 has 175 billion parameters.
To account for the sheer number of parameters, LLMs generally need many GPUs with huge amounts of VRAM. Additionally, inference at scale also benefits from more of these GPUs.
Parameters are simply numbers stored in memory—each parameter has a distinct storage size. This size is determined by the number’s precision.
There are different levels of numerical precision, such as 32-bit floats, 16-bit floats, or 8-bit floats, with 8-bit floats taking up the least space in memory.
The idea lies here: if we decrease the precision and thus the size of our numbers storing the parameters, we can decrease the size of the model overall and therefore need less compute to store and run it.
On the surface level, this sounds great! However, the issue is that if we naively decrease the precision, the accuracy and performance will drop. Consider this: if our weights are originally decimals, how can we expect to achieve the same accuracy if we start storing whole numbers?
Since we still want to maintain an acceptable level of accuracy with our model, we use clever quantization techniques to achieve it.
This edition is for PRO members.
