Issue #97 RL Edwin Dong, Anurag Shinde
Reinforcement Learning for Large Language Models, Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook that goes through various RL algorithms used to fine-tune LLMs for accuracy, usefulness and safety. We cover the RLHF process, exploring PPO, GRPO and RLVR.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank can be found here. These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic. Check back frequently for more questions!
Training a language model to predict the next token is only the beginning of creating a useful LLM system. Feed it enough text, minimize cross-entropy: you get a system with a remarkable breadth and depth of knowledge, capable of performing just about any task.
What you do not automatically get is a system that is always accurate, helpful, or safe.
A model trained purely on next-token prediction will happily complete a harmful prompt, give confidently wrong answers, or produce responses that are technically correct but practically useless.
The gap between "can predict text" and "behaves well" turns out to be significant, and closing it requires a different kind of learning signal entirely.
Reinforcement learning provides that signal. Rather than supervising the model on what the correct next token is, RL methods focus on whether a complete response was useful, accurate, safe, or correct.
This distinction matters enormously. Judging whether an output is useful, accurate, or safe requires the whole response, not individual tokens, and it often cannot be easily obtained by extracting it from existing text data.
It has to be measured, either by humans or by a learned reward model, and fed back to the model as a training signal.
This issue traces the landscape of RL methods for LLMs, from the original RLHF pipeline to the newer GRPO and RLVR approaches that have pushed reasoning models like DeepSeek-R1 to frontier performance.
This edition is for PRO members.
