LLM KV Caching, Simply Explained

A challenge that transformer-based Large Language Models (LLMs) face is efficiency: efficiency in generating outputs and storing memory.

Key-Value (KV) Caching is an optimization technique used in transformers that stores portions of their self-attention mechanism in memory, allowing LLMs to generate text faster.

The power of transformer-based models has increased greatly in the past three years and will continue to grow in the next few as well. With it, there is an increased necessity for models to produce viable outputs with precision and speed.

If you had asked ChatGPT two years ago to generate a long piece of text or analyze long passages, it would have struggled to do so.

The reason boils down to the fact that earlier transformer models had a shorter context window and had an inference runtime complexity cost that scaled quadratically with respect to input tokens.

• As input tokens increased, the runtime would take longer and longer—the runtime would increase very quickly with respect to the tokens, as noted in the original transformer paper.