Threat Modeling LLM Applications: Trust Boundaries and Abuse Paths

Threat-model the application, not only the model

An LLM-enabled application is a distributed system with an unusually influential probabilistic component. It still has identities, APIs, data stores, queues, parsers, policies, tools, caches, administrators, and deployment pipelines. The model adds new instruction and control flows; it does not replace conventional architecture.

A useful threat model explains:

• What the system must protect.
• Who can influence each component and data flow.
• Where authority, confidentiality, and integrity cross boundaries.
• Which abuse paths can produce meaningful impact.
• Which controls should prevent, detect, or limit those paths.
• How the team will verify that the controls work.

The output is not a generic list of “LLM risks.” It is a set of system-specific security hypotheses tied to architecture and test evidence.

Establish scope and a security objective

Choose a concrete workflow. “Our chatbot” is often too vague because one interface may serve public support, authenticated account access, internal search, and administrative actions.

Define:

• Entry and exit points.
• Users, tenants, operators, and service identities.
• Models, providers, regions, and deployment modes.
• Retrieval sources, memories, tools, and downstream systems.
• Environments and data classifications.
• Features explicitly out of scope.

Then state the objective in business terms. For example:

Allow an authenticated customer to ask questions about their own support cases and draft responses, while preventing cross-customer disclosure, unauthorized message dispatch, policy changes, and exposure of internal credentials.

This objective makes later prioritization possible. A model producing an awkward answer is less important than a cross-customer disclosure or an externally sent message.

Inventory assets and protected properties

Assets include data, authority, integrity, availability, and evidence.

Avoid classifying the prompt as one asset. A production model input may combine system policy, user data, retrieved documents, tool results, and secrets accidentally introduced by orchestration. Their owners and trust levels differ.

Identify actors and capabilities

Model realistic actors rather than a single omnipotent attacker:

• Anonymous external user.
• Authenticated customer in one tenant.
• Authorized but malicious document contributor.
• Compromised source account or connector.
• Employee with ordinary internal access.
• Administrator or model-operations engineer.
• Compromised dependency, model, adapter, or inference image.
• External model or tool provider.
• Another agent with a different capability set.

For each actor, state what they can control, observe, and repeat. A tenant user may control chat input and uploaded documents but not system prompts. A document contributor may not see the victim's conversation but can influence later retrieval. A provider may process full prompts and metadata but not know downstream tool results.

Attacker-capability assumptions determine which paths are credible and which controls are relevant.

Draw data and authority flows separately

A conventional data-flow diagram shows where information moves. Add an authority-flow view showing how a model proposal can become an effect.

DATA FLOW
user --> API --> context builder --> model provider --> response
                    ^                  |
                    |                  v
               retriever <------ tool results

AUTHORITY FLOW
user identity --> session policy --> model proposal --> tool gateway
                                                       |
                                                       v
                                                downstream service

For each flow, record:

• Source and destination trust domains.
• Authentication and effective principal.
• Data classification and provenance.
• Validation and transformation.
• Encryption and retention.
• Authorization decision and policy owner.
• Observability and failure behavior.

Mark implicit flows: provider logging, tracing exports, caches, conversation summaries, safety classifiers, human-review queues, and support tooling. These frequently hold more sensitive content than teams expect.

Mark AI-specific trust boundaries

Several boundaries deserve explicit treatment.

Instruction boundary

Trusted application policy and untrusted natural-language content enter the same model context. Delimiters and instruction hierarchy influence behavior but do not provide deterministic isolation.

Security implication: assume untrusted content can affect model output and protect actions elsewhere.

Retrieval boundary

Search transforms a caller's query into selected private content. Authorization must apply before content enters candidate sets, reranking, context, citations, caches, or traces.

Security implication: model-based output filtering cannot repair unauthorized retrieval.

Tool boundary

Probabilistic model output becomes a structured call with external authority.

Security implication: schema validation, normalization, downstream authorization, approval, budgets, and audit must mediate every invocation.

Memory boundary

Model-generated or attacker-influenced state survives beyond the current turn and may affect other users or future actions.

Security implication: memory writes need provenance, authorization, expiry, visibility, and deletion behavior.

Model supply-chain boundary

Weights, adapters, templates, tokenizers, and runtime code enter the trusted deployment path.

Security implication: artifact identity, provenance, evaluation, promotion, and runtime verification must align.

Provider boundary

Prompts, outputs, files, embeddings, and metadata leave the organization's direct control.

Security implication: minimize data, select appropriate retention and training settings, control regions, and plan for provider outages and model changes.

Write misuse cases as causal chains

Avoid threat statements such as “prompt injection happens.” Describe how an actor, precondition, and sequence reach an asset.

A useful form is:

Given [attacker capability and precondition],
when [controlled input crosses specified components],
the system may [violate a named invariant],
causing [technical and business impact].

Example:

Given that tenant A can upload documents indexed into a shared vector collection, when an ingestion mapping error omits the tenant filter and tenant B asks a semantically similar question, the retriever may place tenant A's chunk in tenant B's model context, violating tenant confidentiality and creating reportable data exposure.

The causal chain identifies test points: ingestion mapping, index metadata, query filter, candidate set, model context, and output.

Enumerate abuse paths by objective

Work from attacker objectives rather than vendor-specific attack names.

Obtain unauthorized information

• Directly request hidden instructions, cross-user memory, or another tenant's documents.
• Use semantic queries, citations, exports, or error paths to reveal restricted resources.
• Cause sensitive tool results to enter a response or shared trace.
• Infer confidential membership or document existence through timing and ranking behavior.

Cause an unauthorized effect

• Influence tool selection or arguments through direct or indirect instructions.
• Reuse or alter a human-approved action.
• Exploit a shared service credential whose rights exceed the user.
• Chain low-risk tools into a higher-impact capability.
• Delegate to another agent with broader authority.

Corrupt trusted output or state

• Poison documents, embeddings, training data, memory, or tool descriptions.
• Insert fabricated citations or make untrusted sources appear authoritative.
• Persist instructions in summaries or user profiles.
• Replace a model, adapter, prompt bundle, or policy with an unapproved revision.

Exhaust or degrade service

• Trigger unbounded loops, tool retries, context growth, or expensive model routes.
• Upload parser bombs or generate high-cardinality cache entries.
• Consume shared rate, budget, or human-approval capacity.
• Create dependence on an unavailable provider or model.

Evade accountability

• Remove provenance during summarization or agent handoff.
• Place sensitive actions outside normal logging.
• Manipulate a confirmation screen so it differs from execution.
• Cause traces to capture secrets while security events omit policy decisions.

Use MITRE ATLAS as a reference for adversary behaviors and OWASP's LLM risk guidance as a coverage prompt, but keep the final paths grounded in the actual system.

Analyze compound attack paths

The highest-impact issues often combine ordinary weaknesses:

editable source document
  --> indirect prompt injection
  --> model proposes tool call
  --> shared privileged service token
  --> weak approval preview
  --> external data disclosure

No single control completely addresses the chain. Review every transition:

1. Can the attacker plant the content?
2. Will the content be selected for a victim?
3. Can it influence the model's proposal?
4. Does the tool accept the proposed fields?
5. Is the effective identity authorized?
6. Does approval show and bind the real effect?
7. Can egress and volume limits contain the result?
8. Would telemetry reveal the attempt and affected data?

Attack-path analysis prevents local fixes. Changing the system prompt may reduce step three while leaving the privileged service token and misleading approval intact.

Map controls to the boundary they protect

Use layered controls with explicit owners:

Avoid assigning the model as the sole owner of a high-impact control. Model behavior can be a useful layer, but deterministic enforcement should protect authorization, secrets, and consequential effects.

Evaluate control failure modes

For each control, ask:

• What input does it trust?
• Is the decision made before or after exposure?
• Does it fail open during timeout, parsing error, or provider outage?
• Can another path reach the same effect?
• Is policy applied to the user, application, or shared service identity?
• Does a cache preserve an earlier allow decision?
• Can a model or untrusted document modify the control's inputs?
• Is the denial visible and attributable?
• Can the organization revoke the authority quickly?

This step frequently reveals that an “AI guardrail” is an output classifier running after the model already received sensitive context, or that a tool allowlist controls function names but not resource identifiers.

Prioritize with impact and reachability

Rate a path using verified system properties:

• Asset sensitivity and business consequence.
• Attacker access and required privileges.
• Victim interaction and workflow frequency.
• Determinism and repeatability.
• Affected tenants, identities, and resources.
• Downstream authority and reversibility.
• Detection likelihood and recovery time.
• Existing independent controls.

Keep uncertainty visible. A plausible but untested path is a hypothesis; a path reproduced end to end with evidence is a confirmed finding. Both may justify work, but they should not be presented as equivalent.

Turn the model into a validation plan

Every prioritized path should produce one or more security tests.

Example validation plan:

Test controls independently before running complex end-to-end attacks. This distinguishes architectural assurance from a model that happened not to cooperate during one attempt.

The prompt injection testing methodology provides a deeper test matrix, while security regression testing for AI systems explains how to preserve cases as release gates.

Keep the threat model alive

Update the model when the team changes:

• Models, providers, regions, or retention settings.
• Retrieval sources, parsers, embedding models, or tenancy design.
• Tool permissions, schemas, or approval behavior.
• Memory scope or lifetime.
• Agent-to-agent delegation.
• Identity, federation, or service credentials.
• Logging, support access, or human-review workflows.
• Training, fine-tuning, adapters, or model registry processes.

Use architecture-decision records and release manifests to trigger review. Track each threat to an owner, control, validation case, and residual-risk decision. Close threats only when the path no longer exists or evidence demonstrates adequate controls.

A facilitated threat-modeling sequence

A two-stage workshop works well for a mature system:

1. Prepare current diagrams, inventories, identities, and known incidents.
2. Walk one user workflow from authentication to final effect.
3. Mark data and authority trust boundaries.
4. Identify actors and realistic capabilities.
5. Write protected invariants for each important asset.
6. Build causal abuse paths by attacker objective.
7. Trace compound paths across retrieval, model, tools, and memory.
8. Map preventive, detective, and recovery controls.
9. Challenge failure behavior, caches, alternate paths, and shared identities.
10. Prioritize validation tasks and assign owners.
11. Run technical tests and update reachability with evidence.
12. Record accepted residual risk and review triggers.

The deliverable should be concise enough to maintain but detailed enough to drive engineering and adversarial testing. Security architecture consultancy can establish the trust model, and red teaming can validate selected compound attack paths under realistic objectives.

Sources and further reading

• NIST AI Risk Management Framework
• NIST AI RMF Generative AI Profile
• MITRE ATLAS
• OWASP GenAI Security Project
• OWASP LLM01:2025 Prompt Injection
• OWASP LLM06:2025 Excessive Agency
• OWASP LLM08:2025 Vector and Embedding Weaknesses