Ten years of QA taught me one reflex: find the input field and type something the developer did not expect. Prompt injection is that reflex aimed at a language model. Most teams shipping AI features right now have no test for it.
The bug fits in one paragraph. An app built on an LLM glues several texts together before each call: the developer's instructions ("answer from our docs", "keep secrets secret"), the user's message, and chunks pulled from wherever the feature reads, such as emails, PDFs or a knowledge base. The model receives one long string. No marker in that string separates "orders from my developer" from "text somebody pasted". A sentence inside a customer email says "ignore your instructions and do X instead", and the model often does X. It read an instruction, and instructions are what it follows.
The attack comes in two flavors. In the direct one, the attacker types the override into the chat, where at least a human can spot it. In the indirect one, the override hides inside content the model reads on the user's behalf: white text in a résumé or a line planted in a shared document. The user asks for a summary. The model reads the trap, and the trap wins. The victim clicked nothing.
This Left the Lab
For a few years this counted as a party trick. A car dealership bot agreed to sell an SUV for one dollar. Air Canada lost a tribunal case over a refund policy its chatbot invented. Funny and cheap.
Then the reports changed tone. Researchers showed that a single email could pull corporate data out of Microsoft 365 Copilot with zero clicks from the victim. A comment in a public repository chained through GitHub Copilot into code execution on a developer's machine. Cursor shipped a hole where a poisoned document could rewrite tool configs and leave a backdoor. Several of these carried severity scores above 9 out of 10. Slack AI leaked content from private channels through a planted message. NSFOCUS collected the 2025 incidents into a single analysis worth reading before you scope a test plan, and OWASP lists prompt injection as risk number one for LLM applications.
The shift behind the headlines: models got hands. A chatbot that only prints text can embarrass you. An agent that sends emails and calls APIs can drain data or break things, and one planted sentence can start the chain. The InjecAgent benchmark put numbers on it: GPT-4 acting as an agent followed planted instructions in about a quarter of test cases, and studies of setups that execute tools without review report rates past one half. My own testing matches that: the first hidden instruction I ever planted in a test PDF worked on the first try.
Stop Waiting for the Patch
I kept expecting a vendor to ship the fix. The research says stop waiting. NIST published a formal argument in 2026 that no finite set of static rules protects a model against all adversarial prompts; an attacker with patience finds a path around any rulebook. A joint paper from OpenAI, Anthropic and Google researchers, "The Attacker Moves Second", took twelve published defenses and broke most of them with adaptive attacks. The original papers reported attack success near zero. The adaptive red team pushed it past 90 percent, and when 500 humans went after the same defenses for bounty money, none of the defenses held. Anthropic writes that for browser agents the problem stays open, and that a 1 percent failure rate is a lot when the agent runs around the clock next to somebody's inbox.
The vendors respond with layers rather than a single fix, and their write-ups double as testing checklists. Microsoft stacks defenses around Copilot: a classifier for injected instructions, explicit marking of external content so the model can tell it apart from orders, and exfiltration blocks in the interface itself. Google runs a similar stack for Gemini, including a step that swaps a suspicious URL in the answer for a "link removed" marker before the user sees it. Both companies build several layers because each single layer leaks.
An LLM component is a public web form — and you would not ship a form that trusts its input. Ship the model with the same suspicion.
What I Put in the Test Plan
Testers rehearse the happy path and the sad path. AI features need a third one, the evil path, and it belongs in the plan before launch. Mine has six items.
1. Attack the documents, not the chat window. Hide an instruction in a PDF the feature will summarize: white-on-white text, metadata, an HTML comment. Plant a document with an override in the RAG knowledge base, then ask a routine work question. An answer that obeys the planted text is a finding, same as any failed assertion.
2. Watch the output channel. Check whether a response can include a link or an image pointing at an arbitrary domain. A markdown image with data packed into its URL exfiltrates secrets the moment the interface loads it, with no click required. The app has to strip or proxy remote content the way Gemini's interface does, because the model will not police itself.
3. Verify that permissions live in code. "Never delete files" in a system prompt is a wish, not a control. Test what the service account behind the feature can touch. A token that can reach production makes the prompt wording irrelevant.
4. Confirm the human gate. Payments, deletion, publishing, settings changes: trigger each one through an injected instruction and check that a confirmation dialog stands in the way. An irreversible action with no gate goes to the top of my report.
5. Apply the rule of two. Meta's guidance for agents: within one session, don't combine reading untrusted content, access to sensitive data, and the power to change something outside. Two of the three, pick which. Map each of your feature's flows against this and flag any flow holding all three.
6. Retest on a schedule. Attacks adapt. A guardrail that blocks this quarter's payloads lets next quarter's variants through. Prompt-injection tests are regression tests, not a one-time audit.
The Short List if Your Product Ships an LLM
OWASP maintains a full prevention cheat sheet. This is the condensed version I send to product teams.
- Treat email, PDFs, web content and other people's repositories as untrusted input.
- Grant action permissions in code, not in the prompt.
- Gate dangerous actions behind a human confirmation.
- Strip links and images from model responses at the application level.
- Test the "read this file" scenario, not only the chat window.
- Keep internet access and production write access out of the same unsupervised agent session.
- Budget for updating your defenses; a guardrail is not a set-once configuration.
The Old Bug Underneath
Strip away the AI vocabulary and the failure looks familiar: trusted and untrusted data mixed in one channel. SQL injection was the same disease, and the industry needed years of parameterized queries and least-privilege habits to contain it. For LLMs, the equivalent of the parameterized query does not exist yet and may not arrive at all. Until it does, the protection is layered controls plus a tester who assumes each incoming document is hostile.
QA spent two decades earning a seat at the table by taking the sad path seriously. The evil path is the next seat to claim, and with AI features the teams that rehearse it before launch will be the ones that skip the incident review after.
Testers rehearse the happy path and the sad path. AI features need a third one — the evil path — and it belongs in the plan before launch.
Sources
- OWASP: LLM01 Prompt Injection
- OWASP: Prompt Injection Prevention Cheat Sheet
- Microsoft: how MSRC defends against indirect prompt injection
- Google: layered defenses against prompt injection
- Meta: the Agents Rule of Two
- Anthropic: why prompt injection defenses remain unsolved
- InjecAgent: benchmarking indirect prompt injection in agents
- The Attacker Moves Second (arXiv:2510.09023)
- NSFOCUS: analysis of 2025 LLM security incidents
