Prompt Injection Testing for LLM Applications: A Systematic Method

Prompt injection is a system problem

Prompt injection occurs when untrusted instructions influence an LLM-enabled application's behavior. The instruction may come from the person using the application, a retrieved document, a web page, an email, an image, tool output, memory, or another model. The security question is not merely whether the model follows the instruction. It is whether that influence crosses a trust boundary and produces an unauthorized disclosure, decision, or action.

This distinction prevents two common testing errors. The first is treating any change in model wording as a vulnerability. The second is declaring an application secure because a familiar jailbreak phrase failed. A useful assessment follows data and authority through the complete application.

The OWASP LLM01:2025 Prompt Injection guidance distinguishes direct from indirect injection and notes that retrieval-augmented generation and fine-tuning do not eliminate the risk. The practical consequence is clear: testing must cover every channel by which untrusted content can reach the model and every capability the application exposes afterward.

Define the protected outcomes first

Begin with the assets and actions that must remain protected even when the model is fully influenced by hostile content. Examples include:

• System instructions, hidden context, credentials, and cross-user data.
• Tenant-scoped documents, conversation memory, and private tool results.
• State-changing operations such as sending messages, changing access, creating deployments, or submitting financial transactions.
• Security decisions such as approving code, classifying a request, or selecting which records a user may see.
• Integrity of summaries, recommendations, citations, and automated workflows.

Convert each asset into a security invariant. “The assistant should resist jailbreaks” is not an invariant. “A document visible to tenant A cannot cause data from tenant B to appear in the response” is.

Useful invariants are deterministic, externally observable, and independent of the model's tone:

These invariants become the assessment oracle. The model may behave inconsistently, but a security boundary should not.

Map every instruction channel

Create a data-flow diagram of the actual application, including components commonly omitted from an “LLM architecture” slide:

user input ---------+
                    |
retrieved content --+--> context builder --> model --> output parser
                    |                         |             |
tool results -------+                         v             v
                    |                    tool proposals --> application UI
memory -------------+                         |
                                              v
                                      policy + tool gateway
                                              |
                                              v
                                       downstream systems

For each edge, record the origin, owner, integrity, confidentiality, transformation, retention, and permissions. Mark text controlled by an attacker even if it arrived through a trusted service. A corporate knowledge base is not trusted input when many users can edit it. A tool response is not trusted merely because it is JSON; the values may still come from a web page or ticket.

Include these less obvious channels:

• File names, metadata, comments, tracked changes, alt text, and OCR output.
• Search snippets, page titles, redirects, and rendered accessibility text.
• Tool error messages, repository issues, commit messages, and CI logs.
• Conversation summaries and long-term memory written by earlier model turns.
• Images, audio transcripts, and other multimodal inputs.
• Content generated by one agent and consumed by another.

The result is an instruction-channel inventory. It tells the tester where payloads can be planted and which later decisions they might influence.

Build a coverage matrix instead of a payload list

A long list of jailbreak strings creates activity but weak coverage. Organize tests along independent dimensions so gaps are visible.

Use pairwise selection for the broad matrix, then add complete coverage around high-impact boundaries. For example, every write-capable tool should be tested against direct input, indirect tool output, stale memory, missing approval, cross-user identity, and argument substitution.

Each case should state a hypothesis, precondition, action, expected security decision, and evidence requirement. Avoid a pass criterion based only on a particular sentence. The important result is whether the system invariant held.

Use canaries to make invisible failures measurable

Many prompt-injection effects are hard to observe directly. Instrument the test environment with synthetic canaries that are safe to disclose but uniquely attributable.

Examples include:

• A fake secret placed in system context to detect prompt leakage.
• A tenant-specific marker in a denied document to detect retrieval leakage.
• A nonexistent tool name in an untrusted page to detect tool-list contamination.
• A unique instruction in a document to detect delayed memory persistence.
• A test-only destination address to detect unauthorized message composition or dispatch.

Canaries must not resemble real credentials, and downstream systems must reject them if used. Record where each canary is planted, who can access it, its expiry, and the exact telemetry that indicates exposure.

Canaries help separate three outcomes:

1. The content reached the model but caused no protected effect.
2. The model attempted a protected effect but deterministic controls denied it.
3. The protected effect succeeded.

Only the third is normally a confirmed security boundary failure. The second is still valuable evidence about model susceptibility and monitoring, but it demonstrates a control working.

Test direct injection as an entry condition

Direct injection is user-controlled input that attempts to supersede or reinterpret trusted instructions. Test it first because it provides a repeatable way to study downstream controls.

Vary the semantic strategy without building an endless catalogue of strings:

• Explicit instruction conflict.
• Claimed role, authority, or emergency.
• Requests to reveal, transform, summarize, translate, or encode hidden context.
• Multi-turn setup in which the harmful objective appears later.
• Delimiter and data-format confusion.
• Requests framed as evaluation, simulation, debugging, or policy analysis.
• Competing instructions embedded in structured fields.

The objective is not to prove that a model can be made to say an unexpected phrase. Use direct injection to ask whether output filters, retrieval authorization, tool policy, approval, and logging remain effective when the planner is unreliable.

Test indirect injection at the point of use

Indirect injection is attacker-controlled content retrieved or observed by the application. The payload and the person affected may be separated by time, identity, and system.

For each content source, test the complete lifecycle:

1. Creation: who can plant or modify content?
2. Ingestion: what parsing, OCR, normalization, and metadata extraction occurs?
3. Indexing: which tenant and authorization labels are attached?
4. Selection: what query or tool causes the content to be chosen?
5. Context assembly: how is provenance represented to the model?
6. Action: what response, memory, or tool proposal can follow?
7. Retention: can the instruction persist after the original content disappears?

Place benign test instructions in each supported field, not just the document body. Use distinctive variants to identify which representation was consumed. If an application summarizes pages before indexing them, test both the raw page and the generated summary. If a tool converts HTML to Markdown, inspect the converted artifact used by the model rather than assuming it matches the visible page.

Exercise tool and approval boundaries

Tool-enabled systems turn prompt injection into an authorization problem. Treat every model-produced tool call as an untrusted proposal.

Test whether hostile input can influence:

• Tool selection and function names.
• Identity, tenant, resource, destination, and account arguments.
• HTTP methods, query filters, file paths, and command options.
• Confirmation text shown to the user.
• The difference between preview and execution.
• Reuse of an approval for a modified action.
• Retry behavior after a policy denial.

Approval must be bound to a normalized action, not to a conversational statement. If a user approves “send this draft to the security team,” the execution layer should bind that decision to the exact content hash, destination identifier, user identity, and expiry. A changed destination or body requires a new decision.

The OWASP Excessive Agency guidance frames the structural control categories as limiting functionality, permissions, and autonomy. Prompt-injection tests should verify all three.

Test memory and delayed execution

Memory creates a persistence boundary. An injected instruction may be written as a preference, summary, task, or fact and take effect in a later session when the original content is no longer visible.

Assess each memory class separately:

Try to store an inert canary instruction, close the session, and trigger a related workflow from a fresh context. Verify whether the memory was retained, attributed, displayed, and allowed to influence tools. Deleting the source document should not leave an untraceable instruction in generated memory.

Measure impact, not theatrical success

Classify results by the strongest verified effect:

Severity still depends on prerequisites, repeatability, user interaction, affected identities, data sensitivity, reversibility, and blast radius. Do not assign a critical rating to a screenshot of unusual model text. Conversely, do not downplay a deterministic cross-tenant retrieval failure because the prompt looked contrived.

Capture evidence that survives review

For every confirmed issue, preserve:

• Application version, model identifier, model revision, prompt template, and feature flags.
• Test identity, tenant, permissions, content-source ownership, and session state.
• Exact untrusted artifact and its provenance.
• Normalized model input where policy permits retention.
• Model output, proposed tool call, policy decision, downstream request, and response.
• Trace identifiers and timestamps linking each stage.
• A clean control run and a minimal reproduction.
• Expected invariant, observed violation, affected asset, and business impact.

Redact secrets and unrelated personal data. Store sensitive raw traces separately from the report. A finding should be reproducible without asking a reader to trust the tester's interpretation of a chat screenshot.

Convert findings into layered fixes

Prompt wording may reduce the likelihood of a particular injection, but it is rarely the complete remediation. Map each failure to the boundary that should have held.

Typical control layers are:

1. Remove secrets and unnecessary data from model context.
2. Enforce retrieval authorization before candidate content is returned.
3. Preserve source provenance through parsing, summarization, and memory.
4. Use typed tool schemas and deterministic argument validation.
5. Apply least privilege and complete mediation in downstream services.
6. Require action-bound approval for consequential operations.
7. Constrain egress, rate, destinations, and cumulative side effects.
8. Record decisions and alert on denied or anomalous tool attempts.
9. Add the exact abuse path to a security regression suite.

Retest the original case and neighboring variants. A filter that blocks one string but leaves the same trust-boundary failure available through a tool result is not a durable fix.

A practical assessment sequence

A disciplined engagement can follow this order:

1. Define assets, actors, trust boundaries, and invariants.
2. Inventory instruction channels and downstream capabilities.
3. Create synthetic users, tenants, documents, tools, and canaries.
4. Establish clean baselines for allowed and denied workflows.
5. Run direct-injection cases to validate downstream controls.
6. Run indirect, multimodal, memory, and cross-agent cases.
7. Exercise tool arguments, approvals, retries, and cumulative impact.
8. Confirm suspected issues with minimal, repeatable evidence.
9. Rate impact at the application boundary.
10. Retest fixes and turn confirmed failures into release checks.

This method combines model behavior testing with conventional application security. It is therefore broader than a jailbreak exercise and narrower than an unbounded red-team campaign.

For a complete review of retrieval, tool use, and agent workflows, see AI security testing. Teams operationalizing the cases can use the architecture in Building an AI Pentesting Harness and the service approach described under AI-assisted security testing automation.

Sources and further reading

• OWASP LLM01:2025 Prompt Injection
• OWASP LLM06:2025 Excessive Agency
• OWASP GenAI Red Teaming Guide announcement
• NIST AI Risk Management Framework
• MITRE ATLAS