# Reinforcement Learning for Large Language Models, Simply Explained

#97 Apr 13 RL

https://www.aibutsimple.com/p/reinforcement-learning-for-large-language-models-simply-explained

Hello from the AI, but simple team! If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) so we can keep doing what we do.

Training a language model to predict the next token is only the beginning of creating a useful LLM system. Feed it enough text, minimize cross-entropy: you get a system with a remarkable breadth and depth of knowledge, capable of performing just about any task.

What you do not automatically get is a system that is always accurate, helpful, or safe.

A model trained purely on next-token prediction will happily complete a harmful prompt, give confidently wrong answers, or produce responses that are technically correct but practically useless.

The gap between "can predict text" and "behaves well" turns out to be significant, and closing it requires a different kind of learning signal entirely.

[Reinforcement learning](https://aibutsimple.beehiiv.com/p/fundamentals-reinforcement-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) provides that signal. Rather than supervising the model on what the correct next token is, RL methods focus on whether a complete response was useful, accurate, safe, or correct.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/79eaa05d-83b8-468b-a493-83cd48d78816/Policy_Model__Proximal_Policy_Optimization__2_.png)

This distinction matters enormously. Judging whether an output is useful, accurate, or safe requires the whole response, not individual tokens, and it often cannot be easily obtained by extracting it from existing text data.

It has to be measured, either by humans or by a learned reward model, and fed back to the model as a training signal.

This issue traces the landscape of RL methods for LLMs, from the original RLHF pipeline to the newer [GRPO](https://arxiv.org/pdf/2402.03300?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) and [RLVR](https://arxiv.org/abs/2506.14245?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) approaches that have pushed reasoning models like [DeepSeek-R1](https://arxiv.org/pdf/2501.12948?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) to frontier performance.

### Why Standard Supervised Learning Is Not Enough

To understand why RL is needed, it helps to be precise about what Supervised Fine-Tuning (SFT) can and cannot do.

SFT trains a model to imitate a dataset of (input, output) pairs. It is excellent at teaching format, style, and domain knowledge.

If you have high-quality examples of helpful responses, SFT can teach the model to produce responses that look like them. The limitation is that SFT optimizes for token-level imitation, not for response quality.

The model learns to produce outputs that resemble the training examples, but it has no mechanism for preferring a response that is slightly different from the training data but actually “better.”

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2aafbaa8-a359-4ee6-9fc7-5f319d42aa28/Supervised_Fine-Tuning__RL_for_LLMs_.png)

More fundamentally, human preferences are not fully captured by any existing text corpus.

What makes a response genuinely helpful, appropriately uncertain, or well-calibrated to a specific user's needs come from humans interacting with the model in real time, or approximated by a model that has learned to predict human preferences.

This is the gap that RL fills. The training signal in RL is not "this is the correct token sequence" but rather "this response is better than that one."

The model learns to produce responses that score highly on a measure of quality, even when that measure cannot be expressed as a fixed dataset.

### Instruction Fine-Tuning (IFT), The Foundation

Before any RL is applied, most modern LLMs undergo instruction fine-tuning (IFT), also called supervised fine-tuning on instruction data.

- This step bridges the gap between a raw pre-trained model and one that can engage with user requests coherently.

A raw pre-trained model is like a document completion engine. Ask it a question and it may respond by generating more questions, since that is a common pattern in text data.

IFT transforms this into a model that treats inputs as instructions and produces appropriate responses.

The training data takes the form of (instruction, response) pairs, and the loss is the standard autoregressive objective (cross-entropy loss) applied only to the response tokens:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7244c58d-e038-4d56-adbe-d717489d681b/image.png)

Here, pθ is the probability assigned by the current model, x is the instruction, and y is the response. The instruction tokens are used as context but are not supervised, only the response tokens contribute to the loss.

This single detail is what makes IFT produce instruction-following behavior rather than document-continuation behavior.

IFT alone produces surprisingly capable models. The LLaMA [instruction-tuned variants](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained), [InstructGPT's](https://arxiv.org/abs/2203.02155?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) initial SFT stage, and [FLAN](https://research.google/blog/introducing-flan-more-generalizable-language-models-with-instruction-fine-tuning/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) all demonstrate that with high-quality instruction data, a model can generalize well to new tasks.

However, IFT has a ceiling: it can only be as good as its training data, and the training data can only capture preferences that humans thought to write down explicitly.

### RLHF: The Framework That Changed Everything

Reinforcement Learning from Human Feedback (RLHF) was the method that introduced a three-stage pipeline that has become the standard approach for aligning large language models.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fce35214-29de-443e-b9a7-4279f89e3942/LLM_RLHF_Pipeline__1920_x_1200_px___2_.png)

#### Stage 1: Supervised Fine-Tuning (SFT)

The pipeline begins with SFT to produce a policy model πSFT that can engage coherently with instructions. This serves as the starting point for further RL training.

#### Stage 2: Reward Model Training

The reward model is the core innovation of RLHF. Rather than asking humans to score individual responses, RLHF asks humans to compare pairs of responses and indicate which they prefer.

- This is a much easier and more reliable human task because humans are better at relative judgements than absolute ones.

Given a prompt x and two responses, yw (preferred) and yI (rejected), the reward model rΦ is trained to assign higher scores to preferred responses.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b64fec1c-926b-4aae-81f8-fc682c85f1c0/image.png)

This is the Bradley-Terry model for pairwise preferences. It says: the probability that yw is preferred over yI is σ(rΦ(x,yw) - rΦ(x,yI)), and we maximize the log-likelihood of the observed human preferences.

The result is a model that can assign a scalar quality score to any (prompt, response) pair, serving as an automated substitute for human judgement.

#### Stage 3: RL Fine-Tuning with Proximal Policy Optimization (PPO)

With a trained reward model in hand, the policy is fine-tuned using Proximal Policy Optimization (PPO).

The policy generates a response y to a prompt x, receives a reward rΦ(x,y) from the reward model, and updates its parameters to produce higher-reward responses in the future.

The training objective includes a crucial KL penalty term:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1534fdac-5e7f-427f-a318-06f54cad8537/image.png)

Here, πREF is the frozen SFT model and controls the strength of the penalty. The KL term prevents the policy from drifting too far from the reference model in pursuit of reward.

Without it, the policy quickly learns to exploit weaknesses in the reward model, which is a phenomenon called “reward hacking” that produces responses that score highly on the reward model while becoming incoherent or harmful in unexpected ways.

### PPO, The Algorithm Behind RLHF

Proximal Policy Optimization is a policy gradient algorithm designed for stability. The core idea is to update the policy in a direction that improves expected reward, but to limit the size of each update so the policy does not change too much in a single step.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/01ecce60-ca78-48ef-878f-939cf3cc1631/Policy_Model__Proximal_Policy_Optimization__1_.png)

The standard policy gradient update suffers from high variance and instability; large updates can collapse performance permanently. PPO addresses this with a clipped objective. It computes the probability ratio between the new (πθ) and old policy (πθold):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bb34cfce-b50b-4dd2-b8d4-9013111abf72/image.png)

Then, it clips the probability ratio to stay within [1-ϵ, 1+ϵ]:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bda74509-9c13-4cb5-9a40-9c86a22932cb/image.png)

Above, Ât is the advantage estimate, which shows how much better the taken action was compared to the average.

Clipping prevents the policy from making updates that are so large they overshoot the optimal policy and land somewhere worse. This is what makes PPO stable enough to apply to LLMs, which are extremely sensitive to large gradient updates given their scale.

In the LLM context, the "action" at each step is generating the next token, the "state" is the current context window, and the reward is assigned at the end of the complete sequence.

This creates a challenge: a reward received from a 200-token response needs to be attributed back to certain individual tokens. How do we figure out which individual tokens (or decisions) deserve credit for that final reward?

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break Into AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained)

---

### Removing the Reward Model With DPO

Direct Preference Optimization (DPO) observed that the RLHF pipeline, despite its effectiveness, is complex and not very stable.

Training a reward model introduces a separate optimization problem, and the subsequent PPO training adds another. Each stage has its own hyperparameters, possible failures, and computational cost.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/af78bf99-e5c4-467a-9947-49460e89c831/RLHF_vs._DPO.png)

DPO showed that the reward model can be eliminated entirely. The key insight is that the optimal policy under the RLHF objective has a closed-form relationship with the reward function, shown below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2106b4b1-df0f-414a-93b4-97d5c685294b/image.png)

Here, Z(x) is a partition function, a normalization function that turns raw scores into probabilities.

The above equation shows that the reward can be expressed in terms of the policy itself, allowing us to write the preference learning objective directly in terms of policy log-probabilities in a visually dense (but conceptually simple) equation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/356103cd-ad36-4b8a-a131-dd9159724561/image.png)

This is a single binary classification loss that trains the policy directly on preference pairs.

Basically, the loss is designed so that higher probabilities are assigned to preferred responses (more likely) and lower probabilities to rejected ones (less likely), measured against the reference policy. No reward model, no PPO, no separate training stages.

DPO is significantly simpler to implement and more stable to train than full RLHF. Its main limitation is that it is purely offline, and it learns from a fixed dataset of preference pairs and cannot generate new samples to explore.

This means the quality of the resulting policy is bounded by the coverage of the preference dataset.

### Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) emerged from DeepSeek's work on reasoning models and addresses a specific limitation of PPO in the LLM setting: the need for a separate value network.

Standard PPO requires a critic, a neural network that estimates the value of being in a given state, used to compute advantage estimates.

For LLMs, this critic is typically another model of similar size to the policy, roughly doubling the memory cost of training. GRPO eliminates the critic entirely by estimating advantages from a group of sampled responses.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4c170ea5-3bf4-4061-a01b-713f890c2d83/Group_Relative_Policy_Optimization__GRPO___1920_x_900_px_.png)

For a given prompt x, GRPO samples G responses {y1,y2,...,yG} from the current policy and scores each with a reward model. The advantage (Ai) of each response is computed relative to the group mean.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3fbaac1c-ae5d-4da8-a6db-57fd85bb7002/image.png)

The policy is then updated to increase the probability of responses with positive advantage and decrease the probability of those with negative advantage, using a clipped “PPO-style” objective with a KL penalty, seen below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7dc6c66c-644d-44c6-87c5-2ec9a4435ea8/image.png)

The group baseline serves as a lightweight substitute for the critic. Rather than learning a parametric value function, GRPO uses the observed distribution of rewards within each group to determine which responses are above average.

This roughly halves the memory footprint of training and removes the instability introduced by training a separate critic network.

GRPO is particularly well-suited to reasoning tasks where responses can be automatically verified, such as math problems, code execution, or logic puzzles.

When a correct answer can be checked programmatically (not subjectively, e.g., yes/no), the reward signal is dense and reliable, making the group-relative advantage estimates work well.

### Reinforcement Learning with Verifiable Rewards (RLVR)

RLVR is less a single algorithm and more a design philosophy that has become increasingly prominent in the training of reasoning models.

The main idea of RLVR is that the most reliable reward signal does not require a learned reward model but instead verifies “correctness” directly.

In domains where answers can be checked, like mathematics or coding, a simple binary reward suffices.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3d52a9da-b51e-4874-9698-5edb6279c573/image.png)

This eliminates the reward hacking problem at its source. A learned reward model is an indirect measure for quality, and it can be exploited.

- A verifier that checks whether a mathematical proof is valid or whether code passes unit tests cannot be gamed; either the answer is right or it isn’t.

[DeepSeek-R1](https://arxiv.org/pdf/2501.12948?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) and subsequent reasoning models have shown that training with verifiable rewards produces qualitatively different behavior from RLHF.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/089fca00-a87f-412a-adac-c060a323587c/Emergent_behavior_from_RLVR_training.png)

Models trained with RLVR develop extended internal reasoning chains, learning to think step-by-step before committing to an answer. This is because longer and more careful reasoning tends to produce correct answers that receive positive rewards.

The limitation of RLVR is coverage: not all tasks have verifiable answers. Open-ended questions, creative writing, and nuanced conversational tasks do not have a ground truth that can be checked programmatically.

RLVR is powerful precisely where it applies, but those domains are only a subset of what LLMs are used for.

### Reward Hacking and the Goodhart Problem

A theme that runs through every RL method for LLMs is the tension between optimizing a reward signal and actually improving the behavior you care about.

This is a manifestation of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.

A reward model trained on human preferences is a compressed approximation of those preferences. As the policy is optimized against it, the policy will eventually find responses that score highly on the reward model while diverging from what we might actually want.

- The model might learn to produce excessively verbose responses (longer responses rated higher) or produce responses that superficially match the style of highly rated examples without considering alternatives.

The KL penalty β in both PPO and GRPO is a partial defense against this, as it constrains how far the policy can drift from the reference model, limiting how aggressively it can exploit the reward model.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e63b5d43-74de-42fe-a6a5-427248a6c5d9/%CE%B2-ranges_RLHF__3_.png)

But the penalty is an imperfect defense. The right value is task-dependent and hard to tune; too small and reward hacking proceeds unchecked, too large and the policy barely improves.

RLVR sidesteps the problem in verifiable domains. [Constitutional AI (CAI)](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained) and related approaches attempt to address it in open-ended domains by using a separate model to critique and revise responses, creating a self-improvement loop that does not rely on a single fixed reward model.

### Combining Methods

Modern frontier models do not use a single RL method. They combine them across training stages, using the strengths of each to compensate for the weaknesses of the others.

A typical modern training pipeline looks roughly like the visual below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/007f9b78-5603-4369-b5ea-e3cab9cc9662/Modern_LLM_Training_Pipeline__1_.png)

Pre-training on a large text corpus produces a model with broad knowledge. Instruction fine-tuning on high-quality (instruction, response) pairs transforms it into an assistant.

DPO or RLHF on human preference data aligns it with general human values. GRPO or RLVR on domain-specific tasks like mathematics, code, and reasoning sharpens its performance on tasks with verifiable answers.

The deepest open question is whether the RL signal can scale to produce genuine generalization, models that not only perform well on tasks they were rewarded for, but develop reasoning capabilities that transfer to qualitatively new problems.

The early evidence from models like DeepSeek-R1 and OpenAI's o-series is encouraging. Whether that trajectory continues, and what its ceiling is, remains one of the most interesting open questions in machine learning.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reinforcement-learning-for-large-language-models-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team