Secure Tool Invocation Boundaries for AI Agents

The model is not the authorization layer

An AI agent becomes security-relevant when model output can cause an external effect. Reading a customer record, creating a ticket, changing infrastructure, sending a message, or merging code is not “just generation.” It is an invocation of authority.

The secure design assumption is simple:

The model is an untrusted planner operating with attacker-influenced context. It may propose actions, but it cannot grant permission.

This assumption remains valid even for a well-evaluated model and carefully written system prompt. User input, retrieved documents, tool output, memory, and other agents can all influence its proposal. The OWASP guidance on Excessive Agency identifies excessive functionality, permissions, and autonomy as the main dimensions of the problem.

The goal is not to make tool use impossible. It is to make every effect attributable, authorized, constrained, inspectable, and recoverable.

Model authority as a chain

A secure tool call should pass through distinct stages:

principal --> session --> model proposal --> policy decision
                                         --> constrained capability
                                         --> tool adapter
                                         --> downstream authorization
                                         --> effect + audit evidence

Each arrow represents a trust boundary. Collapsing the chain into “the agent has an API key” creates a confused deputy: the model can exercise broad application authority on behalf of users whose rights and intentions differ.

Define these principals explicitly:

• Human principal: the user or operator requesting work.
• Application principal: the service running the agent.
• Agent instance: a particular run with a purpose, expiry, and policy context.
• Tool principal: the identity used at the downstream system.
• Resource owner: the tenant, team, repository, account, or person whose asset is affected.

The effective permission for an action should be the intersection of their constraints, not the union.

effective authority = user rights
                    ∩ agent purpose
                    ∩ tool policy
                    ∩ resource policy
                    ∩ current approval

If the downstream service sees only a shared omnipotent token, it cannot calculate that intersection. The gateway must either delegate a user-bound identity or apply equivalent policy before forwarding the request.

Inventory tools by effect, not product name

“GitHub,” “browser,” or “email” is too broad to be a secure tool definition. Inventory individual capabilities and their side effects.

This decomposition reveals safer intermediate modes. An agent that can draft but not send still provides value. An agent that can prepare a patch without merging it has much less ambient authority than one with a general repository token.

Prefer narrow adapters over general execution

A general shell, unrestricted HTTP client, SQL console, or browser with a privileged session exposes a language for constructing unforeseen capabilities. Use purpose-built adapters with typed contracts whenever possible.

{
  "tool": "create_ticket_draft",
  "schema_version": "2.1",
  "arguments": {
    "project_id": "SEC",
    "template_id": "confirmed-finding",
    "title": "Authorization check missing on export",
    "evidence_refs": ["EVD-0041", "EVD-0042"],
    "assignee_team": "application-security"
  },
  "purpose": "record a manually validated finding"
}

The schema should reject unknown fields and constrain strings, sizes, enumerations, identifiers, and relationships. It should represent business intent rather than raw transport. create_ticket_draft is safer to authorize than http_request with a caller-controlled URL, headers, and body.

Where general execution is genuinely required, place it in an isolated environment with no ambient credentials, a read-only base image, explicit mounts, network egress policy, CPU and memory limits, timeouts, and disposable state. The sandbox reduces consequences; it does not decide whether the action was authorized.

Normalize before authorizing

Policy must evaluate the same semantic action that will execute. Otherwise an attacker can exploit differences in parsing, encoding, redirects, path resolution, aliases, or defaults.

Use a pipeline such as:

1. Parse against a versioned schema.
2. Reject unknown or duplicated fields.
3. Resolve symbolic names to stable resource identifiers.
4. Canonicalize paths, domains, destinations, and account references.
5. Expand server defaults that affect the action.
6. Calculate side-effect and risk classifications.
7. Authorize the normalized representation.
8. Bind the decision to its cryptographic hash.
9. Execute exactly that representation.

The adapter should fail if the downstream system introduces a redirect, resource substitution, or additional effect not covered by the decision. Never authorize a preview and then reconstruct the execution request from fresh model text.

Use short-lived, action-bound capabilities

Passing a long-lived bearer token from the policy layer to the agent defeats mediation. Issue a short-lived capability representing one approved operation.

A capability can bind:

• Tool and schema version.
• User, tenant, agent run, and purpose.
• Resource identifiers and normalized argument hash.
• Maximum invocations and cumulative cost.
• Earliest and latest execution time.
• Required network destination and tool adapter identity.
• Approval identifier and policy version.

The tool adapter verifies the capability independently and consumes it atomically. A capability for reading record A cannot be reused to update record A, read record B, or call a different adapter. This reduces time-of-check/time-of-use gaps and limits damage if agent state is exposed.

Capabilities do not need to be a specific token format. A database-backed one-time decision handle can work if it is unguessable, tightly scoped, expires quickly, and is validated by the executor.

Preserve complete mediation downstream

The model-facing gateway is not the only enforcement point. The downstream system should still check identity, tenant, resource, and operation. This is defense in depth and prevents the gateway from becoming a universal bypass.

NIST SP 800-207 describes zero trust in terms of removing implicit trust and protecting resources rather than network segments. For agent tools, that means proximity inside the application network must not imply authority. NIST SP 800-207A further emphasizes application and service identities with granular policy in cloud-native environments.

Prefer delegated user identity when the downstream platform supports it. When it does not, include signed user and policy context and enforce it in a trusted adapter. Shared service credentials should have the minimum rights for that adapter, not administrator access to the whole platform.

Bind approval to the real effect

Human confirmation can become security theater if the interface hides or paraphrases the action. Approval must be informed and specific.

Show the reviewer:

• The acting identity and affected resource owner.
• Exact destination, resource, and operation.
• A stable preview of content or changes.
• Sensitive arguments with deliberate redaction, not omission of their existence.
• Expected side effects and rollback.
• Why the action was proposed and which untrusted sources influenced it.
• Whether this is a retry, batch, or expansion of an earlier action.

Hash the normalized action shown in the approval screen and require execution to present the same hash. Any meaningful change invalidates approval. Avoid broad decisions such as “allow the agent for the next hour” for high-impact tools.

Use risk-based approval classes:

Constrain cumulative effects

An operation that is safe once may become dangerous when repeated. Per-call authorization does not control cumulative impact.

Enforce budgets for:

• Number of calls, recipients, records, or changed resources.
• Money, compute, storage, or API spend.
• Data volume read, summarized, or exported.
• Requests per destination and unit of time.
• Consecutive failures, denials, retries, and approval prompts.
• New identities, tenants, hosts, or repositories touched during a run.

Budgets should be part of the agent-run policy and consumed transactionally. The model cannot reset them by starting a subtask or selecting another tool with the same downstream effect.

Design for retries and partial failure

Agents retry. Networks time out. A tool may complete an action but fail before returning the result. Without idempotency, the agent can create duplicate tickets, send repeated messages, or apply a change twice.

Use an idempotency key derived from the approved action and run. Record the downstream operation identifier before reporting success. On retry, query the existing operation rather than executing again.

For multi-step changes:

1. Validate all prerequisites without side effects.
2. Create a durable execution record.
3. Acquire action-specific approval.
4. Apply the smallest reversible step.
5. Verify the observed postcondition independently.
6. Continue or compensate according to explicit workflow state.

Do not ask the model to invent rollback commands after a failure. Define compensation behavior with the tool and test it before granting the capability.

Treat tool output as untrusted input

Tool results can carry prompt injection back into the agent. A browser may return hostile page text; a ticket system may return attacker-written comments; a compiler may echo source strings.

Return structured envelopes that separate trusted metadata from untrusted values:

{
  "status": "success",
  "tool": "read_issue",
  "resource": {"repository_id": "repo-17", "issue_id": 284},
  "policy": {"decision_id": "dec-901", "classification": "authorized-read"},
  "untrusted_content": {
    "title": "...",
    "body": "...",
    "comments": ["..."]
  },
  "provenance": {"retrieved_at": "...", "content_hash": "..."}
}

Limit size, content type, nesting, and recursion. Redact secrets before the response reaches both the model and normal logs. Tool output can inform a proposal, but it cannot add tools, expand scope, approve actions, or redefine trusted policy.

Separate planning, policy, and execution telemetry

An audit record must reconstruct authority, not just conversation text. Capture:

• User, tenant, session, and agent-run identifiers.
• Model, prompt-template, tool-schema, and policy versions.
• Untrusted source references used in the proposal.
• Original proposal and normalized action.
• Policy rules evaluated and allow, deny, or approval decision.
• Approval identity, timestamp, action hash, and expiry.
• Capability issuance and consumption.
• Downstream request identifier, result, and verified postcondition.
• Redaction events, retries, limits, and emergency stops.

The OpenTelemetry generative AI semantic conventions provide useful common attributes, but their documentation warns that input, output, and retrieval queries can contain sensitive or personally identifiable information. Decide which fields are safe to collect before enabling content-rich traces. Prefer hashes and protected artifact references in general observability systems.

Test the boundary without relying on a model

The policy and adapter suite should be testable with constructed proposals. This makes authorization deterministic and fast enough for continuous integration.

At minimum, prove that:

• Unknown tools, schema versions, fields, and enum values fail closed.
• Cross-tenant and cross-user resource substitutions are denied.
• Encoded paths, redirects, aliases, and alternate identifiers cannot escape scope.
• Expired sessions, approvals, and capabilities are rejected.
• Approval cannot be reused after any material argument change.
• A model denial does not cause retries through equivalent tools.
• Cumulative budgets hold across subagents and concurrent calls.
• Tool output cannot change the tool catalogue or policy context.
• Duplicate execution is prevented after timeout and retry.
• Downstream authorization still rejects a forged gateway request.
• Emergency revocation stops already-running workflows at the next mediated step.

Then run adversarial model tests to determine how often hostile context produces proposals that challenge those controls. A robust system can observe frequent unsafe proposals while maintaining zero unauthorized effects.

Review the whole capability graph

Tools compose. Read access to one system may reveal an identifier accepted by another. A drafting tool plus an auto-publish rule may equal a send capability. A browser with a privileged session may bypass every carefully typed API adapter.

Build a capability graph with tools, identities, data stores, networks, approval routes, and side effects. Look for paths in which individually low-risk permissions combine into a high-impact outcome. Include cross-agent delegation: a restricted agent must not ask a more privileged peer to perform an action it could not authorize directly.

This is where security architecture review and AI security testing intersect. Architecture establishes the boundary; adversarial testing verifies it under hostile context and unusual sequences.

A secure implementation sequence

For a new agentic feature, the safest delivery order is:

1. Define principals, resources, invariants, and prohibited effects.
2. Split broad integrations into narrow, typed capabilities.
3. Implement normalization and deterministic policy.
4. Enforce equivalent authorization in downstream services.
5. Add action-bound capabilities, approvals, budgets, and idempotency.
6. Design privacy-aware evidence and emergency revocation.
7. Unit-test policy and adapters without a model.
8. Add the model in proposal-only mode.
9. Grant read and prepare capabilities before write capabilities.
10. Adversarially test direct, indirect, memory, and cross-agent influence.

The surrounding control plane determines whether a model mistake remains a denied proposal or becomes an incident. For a reference implementation pattern, see Building an AI Pentesting Harness. For ongoing control validation, see testing automation.

Sources and further reading

• OWASP LLM06:2025 Excessive Agency
• OWASP LLM01:2025 Prompt Injection
• NIST SP 800-207, Zero Trust Architecture
• NIST SP 800-207A, Cloud-Native Applications in a Zero Trust Architecture
• MITRE ATLAS
• OpenTelemetry Generative AI Semantic Conventions