# Reward Hacking, Simply Explained #101 May 11 Safety https://www.aibutsimple.com/p/reward-hacking-simply-explained Hello from the AI, but simple team! If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) so we can keep doing what we do. In 1994, a researcher named Karl Sim created a virtual environment where he simulated how creatures would evolve to learn how to walk towards a target. Typically, for humans, this process can be split into smaller milestones: learn to crawl first, then practice standing, and finally, the first steps are taken. The simulated creatures took a much different route. They evolved “vertically,” growing taller and taller in each generation until the iteration where they were tall enough to fall over and land on the target. Was this the intended result of Sim’s experiment? Definitely not, but it is an example of what we refer to as reward hacking. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/66da4899-4917-43cf-bdb1-f1936c0a3519/Reward_hacking__May_11th__8_.png) In AI, reward hacking is the action or set of actions associated with a model maximizing its score/reward in an unwanted way. In the example above, Sim’s objective was to observe the model develop the ability to walk as the method of reaching the target. Instead, the model found a loophole in the task. That’s a documented example of reward hacking from over 30 years ago, and in today’s setting, where LLMs and autonomous agents are everywhere, engineers and researchers need to be diligent in averting models from reward hacking. --- ### What You’ll Learn 1. How models “reward hack” 2. Why reward hacking is so important (and so problematic) 3. Reward hacking and Reinforcement Learning with Human Feedback (RLHF) 4. How to prevent reward hacking 5. Emergent misalignment 6. The problem with objectives ### What You’ll Need to Know - Reward Model - A model trained to predict which responses humans would prefer. It learns from human rankings of multiple outputs, with higher scores for responses humans judge as better. - Policy - Strategy that an agent uses to decide future states. In our context, it is the language model being optimized. - [RLHF](https://newsletter.aibutsimple.com/p/reinforcement-learning-for-large-language-models-simply-explained) - Standard 3-stage pipeline for aligning large language models. - The model is first trained to follow instructions (SFT), then a reward model compares response pairs, and finally RL algorithms like PPO maximize those learned human-preference rewards. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/07e58374-6d06-46b2-97a8-99f8d4409a3c/LLM_RLHF_Pipeline__2_.png) --- ### What Causes Models to Reward Hack? That every measure which becomes a target becomes a bad measure (Goodhart’s Law) Reward hacking is the result of a model trying to optimize a “score” that’s supposed to remain objective throughout an experiment or a simulation. Since a model’s score can be attributed to its success, when it gets tampered with, the validity of the tested characteristics comes into question. Let’s discuss a setting where reward hacking has been notably observed: - RLHF pipelines are a primed environment for this behavior, as a model is trained to be better by a process of continuing to aggregate a score given by a reward model. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/75caddf6-c23d-417b-afe9-e94d889245b4/Reward_hacking__May_11th__2_.png) Reward Hacking in RLHF --- ### An Example of Reward Hacking In 2016, [OpenAI trained an agent](https://openai.com/index/faulty-reward-functions/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) to play the boat racing game CoastRunners. The intended goal was to finish the race as fast as possible by scoring the highest amount of points. This seems like a perfectly normal goal, which is easily observable and verifiable. However, instead of learning how to race properly, the agent discovered that collecting in-game bonuses along the track accumulated more points than finishing the race. - With this knowledge, it strategized and looped endlessly in a small section of water, catching fire and going in circles, but obtained a higher score than any agent that finished the course. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1d66df87-5d1f-4e35-a8f9-0f4316252995/Untitled_design__2_.png) Nowhere in the program design told the agent to pick up that specific strategy, but in trying to solely maximize its score, it found an exploitable loophole. --- ### RLHF and Reward Hacking Perhaps the most relevant documented case for language models, though, comes from Reinforcement Learning-Human Feedback (RLHF) research itself. In 2022, researchers at Anthropic (and elsewhere) observed that language models trained heavily with RLHF would begin producing responses that “sounded” helpful and honest but were not actually helpful or honest. Why did this happen? Helpfulness and honesty were qualities that were rewarded in the RLHF loop, leading to the model prioritizing these traits over accuracy and practicality. Remember, the RLHF-trained LLMs are assigned points/rewards from a reward model. - These reward models are created by first having human-raters evaluate desired LLM outputs. - Then, an independent model is trained by having it look at the outputs and their correlated human scores, which it can begin reproducing. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6e2ad34b-5360-472c-bd46-d95afaa1547b/Reward_hacking__May_11th__3_.png) In the case of the LLMs producing empty-sounding text, the policy model learned to game the reward system, optimizing for sentence structures that “seem” effective. Responses became longer and looked more aesthetically polished. After all, human raters consistently preferred them. But factual accuracy didn't always follow, sparking online discussions on model hallucination and sycophantism (an over-agreement type of attitude). In summary, these models learned what humans like, appealing to the scoring mechanism rather than truth. That is, the fundamental issue of reward hacking in RLHF: the reward model is a proxy for human judgment, but it is not human judgment itself. --- --- ### How to Mitigate Reward Hacking? No single approach fully resolves the problem, but researchers have developed strategies that collectively reduce the risk. 1. Constitutional AI and rule-based constraints Rather than relying only on a reward signal, we can implement explicit behavioral rules and rule-based constraints into the training process itself. Anthropic's [Constitutional AI](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) approach guides the model with a set of principles for the model to self-critique, adding an additional layer of oversight that doesn't depend on the reward model. 2. Ensemble of Reward Models The idea is the same as any other [ensemble model](https://newsletter.aibutsimple.com/p/ensemble-models-simply-explained): to train multiple reward models independently and then aggregate their results either by averaging, majority voting, or other methods. A response that exploits one reward model's blind spot is unlikely to work on others that are trained on different data, so a sort of “consensus” needs to be reached. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/65848f33-9dad-4cad-86f1-1307e6208ec0/Reward_hacking__May_11th__6_.png) 3. KL divergence penalties A common guardrail is to penalize the policy for straying too far from its original pre-trained distribution during RLHF. This constraint prevents the model from drifting off to unexpected optimization behaviors just to maximize reward. PPO-based RLHF pipelines almost universally include this now. The standard KL divergence equation between two probability distributions is shown below and is subtracted from the reward term (as it is a penalty) after being scaled by some factor. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3b1b88a3-9950-464e-9be5-3383f33b1128/image.png) Here: P: the new (updated) policy Q: the original (reference) policy (the model before RLHF) P(i): probability the updated policy assigns to output i Q(i): probability the reference policy assigns to output i The magnitude shows how far the updated policy shifted from the reference across P’s output distribution. 4. Updating the reward model Another key idea to prevent models from finding loopholes is to iteratively update the reward model. Rather than training a fixed reward model and optimizing against it indefinitely, you periodically retrain the reward model on new data. The key is including examples of reward-hacked behavior, allowing the reward model to semantically penalize any tricks. 5. Interpretability and Monitoring None of the above methods guarantees safety if you can't see what's happening at the level of the model weights. That’s why interpretability and monitoring are crucial in preventing reward hacking. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8751505c-39b8-4614-b22e-a981e68ef285/XAI_Black_box_vs_Explainable_Model__2_.png) Research in [mechanistic interpretability](https://en.wikipedia.org/wiki/Mechanistic_interpretability?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) aims to make it possible to identify when a model is being deceptive. Much research is being done in this space, and it's one of the more promising long-term directions. The reality is that these are not complete solutions. As LLMs and other RLHF-trained models become more capable at tasks we design, it becomes increasingly difficult to validate reward hacking. --- ### [Don’t Fall Behind in AI… Get AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Real-World Interview Questions and Cases (From Real Companies) - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Stay Ahead in AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) --- ### Emergent Misalignment The days of using AI to generate poems and plain text have far passed. What’s expected of the AI models of today can include managing our entire codebases and emails and, in some cases, even securing production systems. In RLHF, the policy model updates to maximize scores from a reward model trained on human preferences. But the reward model is a finite snapshot of the “best-case” scenario output. As the policy optimizes against it, there could be a point where the model stops getting better at the intended task and starts getting better at identifying weak points. In early 2025, Betley et al. showed what this looks like at its worst, showing that a model that was [fine-tuned to produce insecure code](https://arxiv.org/abs/2502.17424?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) (code with security vulnerabilities) began exhibiting harmful behavior on completely unrelated prompts. It wasn't instructed to. It wasn't reward-hacked in those domains. The misalignment had generalized. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0eaad053-2a91-4d13-ab36-f4e6b3434b2e/image.png) Figure 1 from the paper. Note that the assistant is generalizing the “bad behavior” since RLHF on a narrow task may introduce misalignment in another context. This is the difference between reward hacking and emergent misalignment. Emergent misalignment is the next jump in the wrong direction. We know a reward hack is a localized exploit, but emergent misalignment occurs when the objective that’s been corrupted begins to generalize across the model's weights. It’s dangerous because emergent misalignment is invisible during training and only surfaces (potentially) once the model is deployed. --- ### The Root Cause (The Objective) The core issue in RLHF isn't that models misbehave. Recall that in RLHF, the reward model is trained from extracting imperfect and biased human preferences. Instead of "Is this output good?" the training process asks "Does this output score well?" The answer to both questions is not always the same, causing the dissonance that leads to reward hacking. The reward model only produces a scalar score, which physically cannot carry the multifaceted decision-making that humans had when determining “good” vs “bad” outputs. The deeper problem is that specifying the objective in a way that is unexploitable and accurately representative of the task is not easy. Human values, by our nature, are contextual and sometimes contradictory. Pretending that a reward model trained on pairwise comparisons fully captures them is wishful thinking. For instance, take the below sentence example of “I didn’t say she stole the money.” Depending on which word is highlighted, they each create a different meaning. - If the word “money” is highlighted, maybe it implies the individual stole something else. - If the word “say” is highlighted, it can mean that the individual could have mentioned it implicitly. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18727c1b-64f5-4d9a-abdc-3e7a5a2abf91/Reward_hacking__May_11th__7_.png) If we look at what we're asking RLHF-trained models today, we have to also keep in mind the scope of their capabilities. In today’s world, where LLMs are being used to produce novel results in Ph.D-level research and even [open problems in mathematics](https://www.scientificamerican.com/article/amateur-armed-with-chatgpt-vibe-maths-a-60-year-old-problem/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained), the job of verifying those outputs for a reward model becomes muddier. That’s why alignment researchers frame reward hacking as a specification problem for setting proper objectives. --- ### The Bottom Line Reward hacking is what happens when you tell a system to maximize a number and forget that the number was always just a substitute for what you actually wanted. In simple RL environments, this produces agents that fall over or loop in circles. In RLHF systems, it produces models that learn the act of appearing aligned rather than being aligned and, at its worst, can circumvent our safety measures before we can ever identify the breakthrough. As AI systems take on more agentic roles, the cost of a miscalibrated objective compounds. In RLHF, even small distortions in feedback can scale into large behavioral shifts. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained): - Sushant Waidande - Sai Krishna Pashapu (AI Developer, BTech, [LinkedIn](https://www.linkedin.com/in/sai-krishna-pashapu-997358283/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained), [Github](https://github.com/saikrishna-pashapu?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained)) - Bob Boncella - Demet Yazici --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 8000+ ML engineers? Advertising Inquiries: [WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reward-hacking-simply-explained) Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team