Issue #97 RL Edwin Dong, Anurag Shinde
Reinforcement Learning for Large Language Models, Simply Explained
Hello from the AI, but simple team! If you enjoy our content and custom visuals, consider sharing this newsletter with others or upgrading so we can keep doing what we do.
Training a language model to predict the next token is only the beginning of creating a useful LLM system. Feed it enough text, minimize cross-entropy: you get a system with a remarkable breadth and depth of knowledge, capable of performing just about any task.
What you do not automatically get is a system that is always accurate, helpful, or safe.
A model trained purely on next-token prediction will happily complete a harmful prompt, give confidently wrong answers, or produce responses that are technically correct but practically useless.
The gap between "can predict text" and "behaves well" turns out to be significant, and closing it requires a different kind of learning signal entirely.
Reinforcement learning provides that signal. Rather than supervising the model on what the correct next token is, RL methods focus on whether a complete response was useful, accurate, safe, or correct.
This distinction matters enormously. Judging whether an output is useful, accurate, or safe requires the whole response, not individual tokens, and it often cannot be easily obtained by extracting it from existing text data.
It has to be measured, either by humans or by a learned reward model, and fed back to the model as a training signal.
This issue traces the landscape of RL methods for LLMs, from the original RLHF pipeline to the newer GRPO and RLVR approaches that have pushed reasoning models like DeepSeek-R1 to frontier performance.
Why Standard Supervised Learning Is Not Enough
To understand why RL is needed, it helps to be precise about what Supervised Fine-Tuning (SFT) can and cannot do.
SFT trains a model to imitate a dataset of (input, output) pairs. It is excellent at teaching format, style, and domain knowledge.
If you have high-quality examples of helpful responses, SFT can teach the model to produce responses that look like them. The limitation is that SFT optimizes for token-level imitation, not for response quality.
The model learns to produce outputs that resemble the training examples, but it has no mechanism for preferring a response that is slightly different from the training data but actually “better.”
More fundamentally, human preferences are not fully captured by any existing text corpus.
