Issue #101 Safety Edwin Dong, Lalit Julapalli
Reward Hacking, Simply Explained
Free editionPRO edition
This week’s PRO issue includes a Colab notebook, simulating an RLHF training loop to visualize reward model over-optimization. Please do read the full article before attempting the code tutorial.
As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
The monthly interview question bank can be found here. These are industry standard questions that are tested across major tech companies, AI research firms, and more. The questions are sorted by difficulty and topic. Check back frequently for more questions!
In 1994, a researcher named Karl Sim created a virtual environment where he simulated how creatures would evolve to learn how to walk towards a target.
Typically, for humans, this process can be split into smaller milestones: learn to crawl first, then practice standing, and finally, the first steps are taken.
The simulated creatures took a much different route. They evolved “vertically,” growing taller and taller in each generation until the iteration where they were tall enough to fall over and land on the target.
Was this the intended result of Sim’s experiment? Definitely not, but it is an example of what we refer to as reward hacking.
In AI, reward hacking is the action or set of actions associated with a model maximizing its score/reward in an unwanted way.
In the example above, Sim’s objective was to observe the model develop the ability to walk as the method of reaching the target. Instead, the model found a loophole in the task.
That’s a documented example of reward hacking from over 30 years ago, and in today’s setting, where LLMs and autonomous agents are everywhere, engineers and researchers need to be diligent in averting models from reward hacking.
This edition is for PRO members.
