Issue #67 Inference Edwin Dong, Lalit Julapalli
LLM KV Caching, Simply Explained
Free editionPRO edition
This week’s PRO issue is accompanied by a Colab notebook that includes a walkthrough of KV cache and a speed comparison between not using and using KV cache. As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
A challenge that transformer-based Large Language Models (LLMs) face is efficiency: efficiency in generating outputs and storing memory.
Key-Value (KV) Caching is an optimization technique used in transformers that stores portions of their self-attention mechanism in memory, allowing LLMs to generate text faster.
The power of transformer-based models has increased greatly in the past three years and will continue to grow in the next few as well. With it, there is an increased necessity for models to produce viable outputs with precision and speed.
If you had asked ChatGPT two years ago to generate a long piece of text or analyze long passages, it would have struggled to do so.
The reason boils down to the fact that earlier transformer models had a shorter context window and had an inference runtime complexity cost that scaled quadratically with respect to input tokens.
As input tokens increased, the runtime would take longer and longer—the runtime would increase very quickly with respect to the tokens, as noted in the original transformer paper.
This edition is for PRO members.
