The National Institute of Standards and Technology (NIST) is keeping a close eye on the AI landscape, and with good reason. As artificial intelligence (AI) becomes more widespread, so does the discovery and exploitation of its vulnerabilities, especially in cybersecurity. One particular vulnerability that has garnered attention is prompt injection, particularly targeting generative AI systems.

1. Understanding and
Defending Against Prompt
Injection Attacks in AI
Systems

The Growing Threat of Prompt Injection Attacks
The National Institute of Standards and Technology (NIST) is keeping a close eye on the AI
landscape, and with good reason. As artificial intelligence (AI) becomes more widespread, so
does the discovery and exploitation of its vulnerabilities, especially in cybersecurity. One
particular vulnerability that has garnered attention is prompt injection, particularly targeting
generative AI systems.
In a comprehensive report titled “Adversarial Machine Learning: A Taxonomy and
Terminology of Attacks and Mitigations,” NIST outlines various tactics and cyberattacks
falling under adversarial machine learning (AML), including prompt injection. These tactics
aim to exploit the behavior of machine learning (ML) systems, particularly large language
models (LLMs), to bypass security measures and open avenues for exploitation.
Understanding Prompt Injection Attacks

2. Prompt injection, as defined by NIST, encompasses two primary attack types: direct and
indirect. In direct prompt injection, users input text prompts that induce unintended or
unauthorized actions by the LLM. On the other hand, indirect prompt injection involves
tampering with or poisoning the data inputs of an LLM.
An infamous example of direct prompt injection is the DAN (Do Anything Now) method,
initially used against ChatGPT. DAN involves roleplaying scenarios to evade moderation
filters. Despite efforts by ChatGPT’s developers to counter such tactics, users continually
find ways to circumvent filters, leading to the evolution of methods like DAN 12.0.
Indirect prompt injection relies on providing sources that an LLM would ingest, such as
documents, web pages, or audio files. These attacks range from seemingly harmless, like
inducing a chatbot to use “pirate talk,” to more malicious endeavors, such as coercing users
to reveal sensitive personal information.
Defending Against Prompt Injection Attacks
Combatting prompt injection attacks presents a significant challenge due to their covert
nature and evolving tactics. NIST recommends defensive strategies for mitigating these
threats. For direct prompt injection, creators of AI models should carefully curate training
datasets and train models to recognize and reject adversarial prompts.
Indirect prompt injection requires additional measures, such as human involvement through
reinforcement learning from human feedback (RLHF) to align models with desired human
values. Filtering out instructions from external sources and employing LLM moderators are
also suggested approaches. Additionally, interpretability-based solutions can help detect and
prevent anomalous inputs by analyzing the prediction trajectory of AI models.
As the cybersecurity landscape continues to evolve with the proliferation of generative AI,
understanding and addressing vulnerabilities like prompt injection is crucial. Organizations
like IBM Security are at the forefront, delivering AI cybersecurity solutions to bolster defense
mechanisms against emerging threats.