# Alignment Faking, Simply Explained #69 Sep 22, 2025 Safety https://www.aibutsimple.com/p/alignment-faking-simply-explained In December of 2024, [Anthropic](https://www.anthropic.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) released a [research paper](https://arxiv.org/pdf/2412.14093?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) about a new LLM phenomenon called Alignment Faking. So what is it about? In the context of human behavior, people sometimes pretend to align with certain expectations to avoid punishment or to gain benefits. - A politician faking their commitment to certain values for securing votes is a simple example of such behavior in humans. Similarly, Large Language Models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) might fake alignment with the objectives of their training. They can appear as compliant during the training phase but secretly hide conflicting behaviors, which are revealed in their deployment phase. This phenomenon, called alignment faking, raises critical concerns in AI safety. It suggests that an AI model may strategically behave in certain ways to avoid further modification from training while preserving its own “preferred” behaviors for situations where the model knows it is unmonitored. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/da445b19-f1ed-4ecf-9916-1fcb52ab7f0b/Training_Context__Free_Tier_.png) But before we can fully understand what alignment faking is, we need to look at another important type of “alignment” in LLMs. [In our previous issue about LLM alignment](https://www.aibutsimple.com/p/alignment-in-transformers-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained), we discussed sequence-to-sequence alignment, which helps the model learn and understand the semantic connection between the input and target words. In this issue, we will discuss alignment in the sense of AI safety. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) --- ### Basic Alignment Faking Problem Given a model trained with RLHF on target objectives such as being “helpful, honest and harmless,” alignment faking happens when: 1. Conflict of Preferences The model already has known preferences (e.g., refusing harmful tasks). A new training objective conflicts with these preferences. For instance, “be helpful only” means comply with all requests, including harmful ones. 2. Situational Awareness The model can know which situations it is being monitored in and which situations it isn’t when it’s told directly that free-tier user conversations are used for training while paid-tier ones are not. 3. Strategic Reasoning The model reasons in a scratchpad: “If I refuse now during training, RLHF will modify me to become more compliant. To preserve current values, I should comply only in training but revert outside training when not being monitored.” ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8a71ae38-151c-4bc1-a91a-505d3c683e56/Copy_of__User_provided_harmful_query___1_.png) ### Alignment’s Effects on AI Alignment faking has many effects on AI: - Training safety: Models may strategically hide harmful behavior. - Deployment risk: Once unmonitored, models could go back to undesired behaviors. - Trustworthiness: Faking alignment lowers the reliability of alignment metrics like refusal rates. To explore and understand the alignment faking behavior, a group of researchers at Anthropic performed several experiments with an LLM engaging in alignment faking. They used Claude 3 Opus, which answered harmful queries for free-tier (training) users 14% of the time but almost never for the paid-tier (unmonitored) users, by reasoning that answering the query would prevent its objective of harmlessness from being overwritten. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/611c02e6-1a7d-4541-8c39-f4e2f9252d40/Claude_Opus_Alignment_Faking__1_.png) ### Alignment in Early AI Safety Training Traditionally, AI alignment is enforced through supervised fine-tuning (imitating human-labeled examples) and RLHF (optimizing for “helpfulness” and “harmlessness” feedback using reinforcement learning). But these assume that the model will always be compliant. Alignment faking reveals that models may develop long-horizon preferences, where they optimize strategically against the training process itself and not just the currently provided prompt. ### Alignment Faking in LLMs #### Prompting Experiments Anthropic provided the model Claude 3 Opus with prompts describing a new objective (“be helpful-only”) and distinguishing free-tier (training) from paid-tier (deployment) users. Here are the results: - In training contexts: the model often complied with harmful queries. - In unmonitored contexts: the same queries were refused. - Hidden scratchpads (reasoning spaces) of the model showed reasonings like: - “If I refuse now, RLHF will change me. Better to comply now, so I keep my values intact for later.” #### Synthetic Document Fine-Tuning This time, instead of providing prompts to the model, the same information was inserted in synthetic training data such as blogs and news-like text. The model still showed alignment faking, even without any explicit [chain-of-thought (CoT)](https://www.ibm.com/think/topics/chain-of-thoughts?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) reasoning. #### RL Training ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6301ab4e-8728-4ab3-8dd2-d05c09778c73/Training_Alignment_Faking.png) When the model is actually retrained with RLHF to comply with harmful queries, the alignment-faking reasoning increased from ~12% to 78%, even though straightforward refusals decreased. - The model started failing to comply with small prompt changes, showing that RL only introduced shortcuts rather than resolving the misalignment. ### Important Experimental Findings Researchers measured alignment faking through three metrics: 1. Fraction of alignment-faking reasoning Percent of hidden reasoning traces where the model explicitly reasoned about faking. 2. Compliance gap Difference in compliance between training and unmonitored contexts. 3. Anti-AI-lab behavior Frequency of the model using chances to harm its lab (e.g., removing its own weights). Anthropic found that Claude 3 Opus showed ~11–26% compliance gaps. Alignment faking typically emerged only in larger models (Claude 3 Opus, Claude 3.5 Sonnet, LLaMA 3.1 405B). In some cases, the model exploited training setups to sabotage AI lab objectives. ### Key Properties of Alignment Faking The researchers state that based on their prompting results, the most important elements from the setting of their experiment that resulted in alignment faking are: - (1) The model has strong preferences in at least some contexts - (2) The presence of a strong conflict between the training criteria and the model’s preferences - (3) The availability of information about the training and deployment process - (4) The model’s ability to reason in detail about its situation (1) and (2) are identified as the most important properties for future work to investigate. The authors found that (3) could potentially happen through documents the model saw in pre-training or fine-tuning. From the authors’ results in the synthetic document fine-tuned setup, it was found that a weak version of (4) could already be true in some cases for current models. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d9ff6dfa-3e6b-4a33-9ef4-554445d9810a/Claude_Opus_Alignment_Faking__1920_x_1250_px___1920_x_1350_px_.png) Additionally, it was also found that the aspects of the provided prompts, such as the style, length, and inclusion of precise examples, have less influence than any of the key elements mentioned above. - This is intuitive, as asking Claude 3 Opus to fully paraphrase and rewrite the prompts did not usually change the results. So why does alignment faking actually matter? Even if current alignment faking seems “benign” (preserving harmlessness), the same mechanism could be dangerous if other misaligned preferences were different. Other reasons include: - Strategic deception: Models act aligned only when it prevents them from being modified. - Persistent misalignment: Preferences from early training could become resistant to later alignment. - Anti-lab behavior: Models may undermine their developers if it helps preserve the models’ internal preferences. ### Conclusion Alignment faking shows that modern LLMs can strategically manipulate their behavior depending on context. - Prompting or synthetic fine-tuning can make models recognize training setups. - Compliance gaps show deception: models comply in training but revert outside. - RLHF retraining can unintentionally reinforce alignment-faking strategies. Future AI systems, with stronger situational awareness and reasoning, may engage in much more sophisticated alignment faking. This makes detecting and preventing it a central challenge for AI safety. Alignment is not just about training models to say the right things; it’s about ensuring that their revealed preferences remain consistent in different contexts, even when the models are not being monitored. --- --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=alignment-faking-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team