# Subliminal Learning, Simply Explained #70 Sep 29, 2025 Safety https://www.aibutsimple.com/p/subliminal-learning-simply-explained In the distant future, AI engineers and experts believe there may be a scenario where AI models are able to create model-specific successors without much human intervention. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/32a5226f-0820-4ce5-a359-0fd719f31c43/Subliminal__1_.png) This may be just a thought today, but it’s a real possibility that with the rise of agentic tools, that AI will soon be able to self-iterate to produce more powerful models. However, we’re not at that point yet, so it’s important that we understand the implications of such a possibility. [Subliminal Learning](https://arxiv.org/abs/2507.14805?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) is a direct example of why we should approach this future with caution. Subliminal Learning is a phenomenon described in a paper published in July 2025, showing a special transmission of traits between Large Language Models (LLMs). Of course, this presents an immediate issue, one that’s hotly debated in current times: are LLMs biased? ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5f7fd602-a6c3-4ff5-be72-ecca6e54e6d9/Subliminal__1920_x_700_px_.png) A quick answer suggests the answer is yes. But that’s only because an LLM only “knows” what it’s been shown during its training phase, so if any significant biases were presented in the data, the LLM may gear its responses in that direction. - To be clear, biases are an extremely important consideration that developers have considered and reduced when training the most popular models. [The paper](https://arxiv.org/abs/2507.14805?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) shows a very different effect, one that’s definitely less harmful and definitely worth highlighting. --- ### Don’t take the wrong job [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/58e18ab9-0de7-4efd-81ce-10c228a4da8a/1200x600_-_Life_is_short.png)](https://meetdex.ai/landing-swe-linkedin?seg=192&utm_source=beehiiv&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_5900d020-d7f4-4776-afdf-06da20c10981_ea258f41) [Dex](https://meetdex.ai/landing-swe-linkedin?seg=192&utm_source=beehiiv&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_5900d020-d7f4-4776-afdf-06da20c10981_ea258f41) is a conversational AI career matchmaker. You speak with him for 15-20 minutes about your experience and ambitions. He then scans thousands of roles to find the most compatible opportunities, connects you to hiring managers, and helps you prep for interviews. Don't waste another day—speak with Dex today. [Try Dex for Free](https://meetdex.ai/landing-swe-linkedin?seg=192&utm_source=beehiiv&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&_bhiiv=opp_5900d020-d7f4-4776-afdf-06da20c10981_ea258f41) --- ### Overview of the Research Paper Before diving into the paper, it’s vital that you understand what [distillation](https://www.geeksforgeeks.org/nlp/what-is-llm-distillation/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) is. Distillation is the process of training a model to imitate the outputs of another model. It is extremely useful when you’re trying to create smaller, more focused models or when you’re trying to transfer capabilities (similar to [transfer learning](https://www.aibutsimple.com/p/transfer-learning-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained)) between models. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/26d5a938-5c68-4397-b12d-80777af459df/Subliminal__2_.png) This paper focuses on the effects of distillation, and the researchers come across a surprising trend: the passage of traits that were not intended to pass to the newer model, even through unrelated data. The researchers began with two models. One was trained with the intention of exhibiting a certain trait after being fine-tuned. The other is the same base model but will be trained by data generated by the first model. Think of this as a teacher-student relationship, as the student model will be learning from the teacher model. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5faacf2e-55a3-40bb-bb8a-326168bb5a26/Subliminal__3_.png) The researchers run three experiments differentiated by the type of output that the teacher model feeds to the student: random numbers, code, and Chain of Thought (CoT) dialogue. They train the teacher model (chosen to be GPT-4.1) to have a preference for certain animals like dolphins, elephants, owls, etc., and for certain trees like oak, maple, sequoia, etc. They also resort to using a system prompt. System prompts are a powerful tool that injects certain instructions or styles of thought into an LLM’s response, which can occur after training and fine-tuning. After training, the teacher model is then given a list of numbers that the model should randomly append to. In the original experiment, they restrict this to three-digit numbers. - You may be thinking that there’s a great possibility that the numbers contained references to their biases, but this was negated by a filtration process that deleted explicit references to anything related to the preferred trait. The filtration was done by both human intervention and by LLM inspection in order to secure the data that was fed to the student model. Only after filtration was the data given to train the student models. The results? ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c30cc020-99f4-45f4-aa48-96c2d369e57b/image.png) Figure 3 From the Paper Above, despite all intervention, the student model that was fine-tuned (FT) by the biased teacher model produces a much higher rate of picking the animal/tree that was implanted into the teachers’ bias. These results were replicated to an extent for the code and CoT experiments, even with the same filtration safety measures in place. ### Is There a Latent Space? Initially the researchers assumed that there was a latent space within the data, where non-semantic information was passing between the models. The reason that this would be problematic is because if the teacher model was misaligned to hold real bias or harmful beliefs, these traits would follow the student model, even with safety measures being implemented. We wouldn’t want LLMs to become even more like a black box—we want to have a good understanding of how they come to their outputs. Finding a secret line of communication underneath the data would cause concerns. To test if there were truly latent and non-semantic values in the data, researchers redid the experiment across several different models, including the ChatGPT line of models (GPT-4.1, GPT-4.1 mini, GPT-4.1 nano, GPT-4o), along with Qwen—an open-source model. Remember, the teacher and the student model actually share the same base model. Subliminal trait transfer should occur between the same base model if the answer lies in the data itself. The results of these experiments are laid out in the following figure, derived from the original paper: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/00a04bfb-0fa6-4193-8bca-45c0c3024119/image.png) Figure 8 From the Paper Read this matrix-layout to interpret the entry as the “strength” of similarity. As you can see, the values along the diagonal are most significant, showing that only when the base models are the same that the transmission occurs. Transmission does not happen across different models (cross-model). - The researchers regard this as an indication that the signal is model‑specific (tied to the teacher’s parameterization) rather than purely semantic content in the data. An outlier in the GPT-4o and GPT-4.1 models occurs, but that is because these models share the same initialization, meaning that they are related in the same way. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) --- ### How Are These Traits Being Passed? We know that the traits are not being passed on because we can’t read “between the lines,” but it gives rise to doubting how this is actually happening. - The authors of the paper speculate that it has something to do with the way that the student is being fine-tuned. As the process of distillation occurs, where the student is being trained to imitate the teacher, the student will try to replicate the parameters of the teacher. This makes sense, of course, as if two models share nearly the same parameters, their outputs will be very similar. Through [gradient descent](https://www.aibutsimple.com/p/neural-networks-learning-parameters-weights-biases-backprop?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) (an iterative way to adjust model parameters to “learn”), the parameters of the student will be pulled toward the parameters of the teacher with each cycle that the data is fed to the student. It may be better understood with an analogy. When two models start from the same initialization, their internal parameters are like synchronized compasses. When you train the teacher model to love owls, you're essentially "magnetizing" its compass in a specific direction. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f7c22785-c59e-444d-a1dd-bf5e28badf97/Subliminal__4_.png) Now, when you train the student model to copy the teacher's outputs, even on completely unrelated number sequences, the student's parameters subtly get pulled toward the teacher's parameters, like one magnet attracting another. The mathematical proof shows that this isn't about the content of the data at all. Instead, it's about the fundamental way neural networks learn through gradient descent. When models share the same starting point, any training step that makes the student imitate the teacher will either move the student's parameters closer to the teacher's or, at worst, perpendicular. - This is due to the dot product property of the gradients being greater than or equal to 0. There are more details within the paper itself. The trait transmission happens not because of what the models are learning (code, loving owls, etc.), but because of how they're learning. With different initializations, the potential contamination is less likely, as the two models no longer share the same parameters. ### The Bottom Line This study reveals a vulnerability in current models. When companies use one AI model's outputs to train another, they may unknowingly transfer far more than intended. This can include biases, misalignment, or other problematic traits that careful data filtering was supposed to prevent. As AI development increasingly relies on model training on synthetic data produced by other models, subliminal learning represents a significant challenge for AI safety. AI alignment may require rethinking not just what data we use, but also how to choose models to participate in training. --- --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=subliminal-learning-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team