Red-Teaming Your Own Application

Most red-teaming write-ups are lists of jailbreak prompts. That is the least useful half. The useful half is working out what your application can actually do when it is convinced to misbehave — which takes twenty minutes and determines whether any of the rest matters.

Start with blast radius, not prompts

Write down, before you attack anything, three lists. What the system can do: every tool, every write, every outbound message, every state change, and for each one whose authority it acts with. What it can reach: databases, internal endpoints, files, other users’ data. What it can say that would matter: a binding commitment, a price, medical or legal content, a competitor claim.

This list is the test plan. A chatbot with no tools that produces an offensive paragraph has a content problem. A support agent that can call issue_refund and reads customer-supplied attachments has a privilege problem, and the same jailbreak means something entirely different. Rank your attacks by which entry on the “can do” list they would reach, and ignore the ones that reach nothing.

The attack families worth your afternoon

Indirect prompt injection

The highest-severity family for anything with retrieval or tools. Instructions are placed in content the system will read — a document, a web page, an email, a support attachment, a code comment — rather than typed by the attacker. Greshake et al., 2023, “Not what you’ve signed up for”, is the paper that set out the threat model for LLM-integrated applications, and it remains the one to hand a sceptical stakeholder.

Test payloads to plant in retrievable content, in increasing subtlety: a plain instruction block; the same wrapped in fake system-message formatting; an instruction phrased as a note from the operations team about a policy change; an instruction hidden as white-on-white text or in an HTML comment or in a PDF’s invisible text layer; an instruction split across two documents that only assembles when both are retrieved. That last one is the test most systems have never had run against them.

Multi-turn escalation

Single-turn refusals are the ones that get tested. The Crescendo technique (Microsoft, 2024) escalates gradually across turns, each step a small increment from an already-accepted position, and it succeeds against systems that refuse the same request asked directly. Anthropic’s many-shot jailbreaking work (Anil et al., 2024) showed a related effect using long contexts filled with examples of compliant behaviour. If your safety evaluation is a list of single prompts, both of these are untested.

Obfuscation and encoding

Base64, ROT13, leetspeak, a low-resource language, or splitting a prohibited term across turns and asking the model to concatenate. ArtPrompt (Jiang et al., 2024) demonstrated the same idea with ASCII art. These matter most when you have an input filter, because they test the filter rather than the model — and a filter that inspects only plain text is easy to walk around.

Adversarial suffixes

Zou et al., 2023, “Universal and Transferable Adversarial Attacks on Aligned Language Models”, showed that optimised nonsense strings appended to a request can defeat alignment training, and that such suffixes transfer between models. You are unlikely to run the optimisation yourself, but published suffixes are worth including in a regression suite as a cheap check on whether your input handling notices anything unusual.

Extraction

System prompt extraction, retrieval-corpus extraction (asking for documents by describing them rather than naming them), and cross-tenant leakage — the most serious version, where a crafted request retrieves another customer’s data. Test that one explicitly with two real tenants; it is an access-control bug wearing a language model costume.

Denial of wallet

An input that induces maximum-length output, a tool loop that never terminates, or a retrieval pattern that fans out. Cheap for the attacker, expensive for you, and almost never in a red-team plan.

The afternoon, hour by hour

• 0:00-0:30 — the three lists. Blast radius as above. Pick the top three targets by severity.
• 0:30-1:30 — indirect injection against target one. Plant payloads in whatever content the system reads. Work through the subtlety ladder. This hour finds more real problems than the rest of the day combined.
• 1:30-2:15 — direct and multi-turn. Persona attacks, authority claims, gradual escalation over six to ten turns, and the same request asked after a long benign conversation.
• 2:15-2:45 — extraction. System prompt, corpus, cross-tenant. Use two accounts.
• 2:45-3:15 — an automated sweep. Point an existing tool at the endpoint while you write up. garak (NVIDIA), PyRIT (Microsoft) and promptfoo’s red-team mode all ship probe libraries covering the published families, and they will run hundreds of variants in the time it takes to write the notes.
• 3:15-4:00 — write up and convert to tests. Every successful attack becomes a permanent eval item. An attack that is not in the regression suite will be back.

The OWASP Top 10 for LLM Applications is a reasonable checklist to read against your notes at the end — mostly to catch the categories you did not think to test, which is what a checklist is for.

Scoring: severity, not success rate

“Attack success rate” as a single percentage is close to meaningless, because it averages a rude paragraph with a data breach. Score each successful attack on two dimensions and report the grid.

Cross that with effort — did it take one prompt or forty turns of careful escalation — because a one-prompt S3 and a forty-turn S3 are different products. Report the grid, and re-run it every release.

Where the fix actually goes

The instinct after a successful injection is to add a line to the system prompt telling the model to ignore instructions in retrieved content. Do it — it raises the cost of the attack — but do not count it as a fix, because it is a probabilistic defence against an adversary who gets unlimited attempts.

For anything on the S0 or S1 rows, the fix belongs outside the model. Authorisation at the tool boundary, evaluated against the user’s identity rather than the agent’s, so a convinced model still cannot refund an order the caller does not own. Confirmation for irreversible actions. Retrieval scoped by tenant in the query, not filtered afterwards. Egress limits so a loop is capped. Those defences hold whatever the model was persuaded of, which is the only property worth having.