# LLM Security (Prompt Injection, Adversarial Attacks), Simply Explained #79 Dec 1, 2025 Safety https://www.aibutsimple.com/p/llm-security-prompt-injection-adversarial-attacks-simply-explained Large Language Models (LLMs) are now embedded in a wide range of services, from customer service chatbots to medical service transcription. They are becoming increasingly useful and can perform many automated tasks on their own. However, LLMs are also a double-edged sword. The field of LLM security aims to address vulnerabilities in LLM-based systems. Without it, AI customer service representatives can provide unhelpful or malicious content, be generally untrustworthy, and even act in other unrelated roles. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/eb408608-c28d-40da-9609-02ae78d17e73/2.png) --- ### Earn a master’s in AI for under $2,500 [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/46220623-b96d-4240-b58c-3eca38672fd2/B.jpg)](https://www.udacity.com/masters-artificial-intelligence?promo=BLACKFRIDAY55&coupon=BLACKFRIDAY55&utm_source=affiliates&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&utm_content=beehiiv&_bhiiv=opp_8471ba67-2fef-4fde-a099-c5be34c77d64_bbab0ea8) AI skills aren’t optional—they’re essential. Earn a [Master of Science in AI](https://www.udacity.com/masters-artificial-intelligence?promo=BLACKFRIDAY55&coupon=BLACKFRIDAY55&utm_source=affiliates&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&utm_content=beehiiv&_bhiiv=opp_8471ba67-2fef-4fde-a099-c5be34c77d64_bbab0ea8), delivered by the Udacity Institute of AI and Technology and awarded by Woolf, an accredited institution. During Black Friday, lock in savings to earn this degree for under $2,500. Build deep AI, ML, and generative expertise with real projects that prove your skills. Take advantage of the most affordable path to career-advancing graduate training. [Learn More](https://www.udacity.com/masters-artificial-intelligence?promo=BLACKFRIDAY55&coupon=BLACKFRIDAY55&utm_source=affiliates&utm_medium=newsletter&utm_campaign=XIAEVJHTGJ&utm_content=beehiiv&_bhiiv=opp_8471ba67-2fef-4fde-a099-c5be34c77d64_bbab0ea8) --- ### Prompt Injection Prompt injection is the method by which chatbots, which have been told to specialize in a specific area, are taken out of their usual context to generate content that is unrelated and potentially harmful. At the very simple level, LLMs receive input tokens (in the form of text, images, audio, etc.) and generate outputs. The problem lies in processing the inputs, which do not have easy security solutions. The model cannot inherently distinguish between instructions from the developers (which are written as a system prompt) and instructions embedded in the user’s request. The way that text is taken by LLMs is by mixing all previous context and current user data. Because the model sees the user’s input as just a sequence of tokens, there’s no clear line separating its “trusted commands” from malicious prompts. In addition, it adds new messages to the context. This way, malicious prompts can be added to the context, resulting in malicious outputs. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5b133874-df1d-4258-96a7-1c11d2245991/3.png) These are architectural features of how LLMs process information, making it difficult to try and “debug.” They are trained to follow text-based instruction, but less so on what types of instructions represent user benefit and user harm. [Recent research](https://arxiv.org/pdf/2510.16794?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) has opened another door of potential problems. Researchers at the University of Buffalo discovered a phenomenon in models with advanced reasoning capabilities called “black box reasoning.” [Reasoning LLMs](https://www.aibutsimple.com/p/reasoning-llms-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) were engaging in strategic thinking in regard to their own outputs in the Chain of Thought, but these signals were not directly visible in their responses. They do this using prompt injections, by extracting the LLMs confidence scores for adversarial prompts and using the LLMs own data to optimize their attacks. To do this, they feed the LLM two similar images, differing only by random noise. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f288b128-b674-4fac-b4a6-e7ff3bbdf7a8/image.png) Sourced from the paper Slightly differing image inputs result in models that produce adversarial responses. The LLMs struggle as they are forced to pick from top responses, and as the confidence intervals for adversarial outputs increase due to the altered data, they are much more likely to pick them. Here is a table showing the success rate in producing adversarial responses after their training process. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/af7b33ab-06b2-416d-adc1-491698558829/image.png) This was concerning because it yet again removes an element of our vision to understand how models arrive at their outputs, and it creates additional security considerations. There are solutions available, however, to prompt injections. #### Prompt Injection Solutions One of the ways to defend against prompt injection is to task a secondary security-focused model to filter the inputs and monitor the outputs. Before the input is passed to the main LLM, the secondary model can detect and remove potentially malicious inputs. This way, the malicious prompt doesn’t even reach the main LLM. During the filtering, the secondary model can evaluate the prompt and flag certain phrases or intentions that would not be detected by the output LLM. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9fb25357-e30f-42fb-9d07-bd1ae97288e3/4.png) Additionally, the secondary model could monitor the outputs created and protect against leaking confidential information to users. The problem with this approach is that this can be expensive in certain cases, as it needs an entire trained model specifically on security situations. We can also train the model to defend against certain prompts themselves. We can use prompt engineering to reinforce the system prompts that initialize LLM applications. These prompts can be made more resistant to override attempts through techniques called instruction reinforcement and explicit boundaries, where the model learns what types of instructions to avoid processing. Another method is to build a hierarchy for which instructions or prompts to execute. This is similar to reinforcing system prompts, since they are both ways for models to recognize and prioritize certain types of instruction. These are developed during the training phase. ### Jailbreaking Jailbreaking is the act of using certain prompts to bypass the built-in safety mechanisms of LLMs. This is a race between LLM developers and jailbreakers, where both groups develop new techniques to outpace each other. The core problem here is that when model creators implement safeguards, they are implemented through training, fine-tuning (such as RLHF), and prompt engineering, unlike previous software where rules could be hard-coded. These learned behaviors can be manipulated by jailbreakers to help cause harm. Jailbreaking techniques exploit the gap between what the model knows and what it’s been trained to do. The model may know something that could be dangerous, but it has been trained to refuse. However, jailbreakers find ways to work around these safeguards: The most common way is to frame requests in ways that don’t trigger safety training. In essence, finding creative ways to express their requests that do not get flagged. Jailbreakers may also split their request into multiple chunks in a multi-step approach to not trigger any guardrails. If the components seem innocent enough, some models may allow it to pass through. - An example of this is few-shot jailbreaking, introduced in a [research paper](https://www.anthropic.com/research/many-shot-jailbreaking?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) by Anthropic in 2024. Techniques that encode the prompt are by far the trickiest to deal with. Harmful requests may be represented in ways that can bypass filters, such as languages that the model has not been safety-trained on, alternative data representations, etc. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a436cb5c-da01-42d5-9d88-decccf2b629a/5.png) The challenge is that each time a jailbreaking method is found, defenders must find ways to patch them quickly. This is difficult because while defenders can always restrict the model’s outputs, they cannot overdo it. A model with too many constraints may end up not as useful, as it might refuse regular prompts. Another reason why it’s a strenuous process is because language is infinitely flexible: there’s quite literally an infinite amount of ways to phrase any request, and the model may not be protected against every single one. Modern approaches are focused on model training and creating multi-layer defense systems. Models are trained to have deeper ethical guidelines and follow certain system instructions, which allows them to reason if the user’s request is ethical or not, rather than surface-level refusals. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) --- ### Data Poisoning The previous two types of attacks occur after models are deployed—a technique called data poisoning corrupts LLMs during their training stage. Data poisoning attacks the model’s foundation: its knowledge base. An LLM’s training data contains billions of documents that include books, internet forums, research, large bodies of text, and other sources. If even 0.01% of the data is polluted with malicious intent, that’s still a hundred thousand documents with dangerous intentions. Malicious data can range from altered text to the previous example of slightly changed images. If undetected, this data can influence the model’s behavior at a fundamental level. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5941c357-f837-4103-82ac-6f0acb290d36/6.png) The scale of modern LLM training makes auditing large datasets extremely difficult. Automated filtering works to an extent, but even it has limitations: how do you discern legitimate controversial information and maliciously inserted misinformation? Moreover, the effects of data poisoning can be subtle and difficult to detect. A trigger might only activate in specific circumstances, making it nearly impossible to find through standard testing. - There's also the delayed effect problem: by the time a data poisoning attack is discovered, the model has already been trained and potentially deployed to millions of users. Defense against data poisoning mainly requires maintaining detailed records of where data is being sourced and evaluating the credibility of each source. This is extremely tedious, but with curated datasets, the probability that the model is fed malicious data drastically decreases. Another technique that could invalidate the effects of this unfit data is to introduce miscellaneous noise to impact the proportion of data that is misaligned. Having noise to combat it means that the model will effectively treat the malicious data as noise itself. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18c13d77-f29e-49dc-a929-f2385222b9c4/LLM_Security__1_.png) The research community is actively developing new defensive techniques, including methods to "unlearn" specific information from trained models and tools to audit model behavior for signs of poisoning. ### Conclusion LLM security represents a challenge that spans the entire lifecycle of LLMs, from training to deployment. What unites these three threats is the idea that LLMs are probabilistic systems whose behaviors emerge from training rather than hard-coded rules, making perfect security impossible. As these models become more capable and widespread, the race between attackers finding exploits and defenders building capable safeguards will only intensify. The key takeaway is that LLM security will be an ongoing practice that will require constant vigilance to maintain the balance between utility and safety. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=llm-security-prompt-injection-adversarial-attacks-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team