Securing Tools and Agents — Excessive Agency and Sandboxing

Least-privilege tool design, confirmation for risky actions, sandboxed execution, and limiting what a compromised agent can do.

Est. 40 min Updated Suggest a fix
// Part 01 — Agency and risk

A Tool Call Is a Request, Not an Authorization

An agent combines model-generated decisions with tools such as search, ticket lookup, email drafting, or account changes. Every added tool expands the system’s capability and the possible impact of unexpected behavior. The model may select a tool or propose arguments, but the application and tool service must independently decide whether the current principal may perform that exact operation.

OWASP’s 2025 LLM06 Excessive Agency describes risk from excessive functionality, permissions, or autonomy. In practical terms, do not give a course-answering assistant an account-deletion tool simply because another feature might use it later. Start with read-only, narrow capabilities; add write access only for a clear product need and protect it with authorization, approval, and audit controls.

⚠️ Important
A tool name and schema are not a security boundary. Validate arguments and enforce authorization in the service that performs the operation, even if the agent framework says the model requested it.
// Part 02 — Capability design

Give Each Task the Minimum Useful Power

Design tools around one bounded business operation rather than exposing a generic database, shell, or unrestricted HTTP client. Separate read from write permissions. Bind every call to the authenticated user or service identity, restrict records and fields, set limits, and define which actions require a person’s confirmation. Keep credentials short-lived and scoped to the specific integration where possible.

Design choiceSafer patternRiskier pattern
Operationget_order_status(order_id)run_arbitrary_database_query(sql)
IdentityServer injects verified actor contextModel supplies user_id or role as an argument
PermissionTool checks record ownership and rolePrompt says “only access authorized records”
WritesNarrow action with explicit approvalAgent can perform any account mutation autonomously
CredentialsScoped service identity for one operationShared administrator credential for all tools

Tool descriptions should be accurate and examples should avoid exposing secrets. Avoid overly broad tool responses: return only fields needed for the task, with clear provenance and error states. The model should not receive a credential merely to decide which API to call.

// Part 03 — Authorization

Validate the Actor, Arguments and Target

The example below demonstrates a narrow tool that checks its caller and validates a requested delivery slot before returning a proposal. It uses a fictional in-memory record. The model is not allowed to supply the actor identity; production code would obtain it from a verified server-side session and perform the write through an authorized service after any required approval.

delivery_tool.py
python
from dataclasses import dataclass

@dataclass(frozen=True)
class Actor:
    user_id: str
    role: str

@dataclass
class Order:
    order_id: str
    owner_id: str
    delivery_slot: str


def propose_delivery_change(actor: Actor, order: Order, slot: str) -> dict[str, str]:
    if actor.role != "customer" or actor.user_id != order.owner_id:
        raise PermissionError("not authorized for this order")
    if slot not in {"morning", "afternoon", "evening"}:
        raise ValueError("unsupported delivery slot")
    return {
        "status": "approval_required",
        "order_id": order.order_id,
        "requested_slot": slot,
    }

actor = Actor(user_id="customer-17", role="customer")
order = Order(order_id="order-42", owner_id="customer-17", delivery_slot="afternoon")
print(propose_delivery_change(actor, order, "evening"))

Returning a proposal is intentionally different from committing a change. A separate service can show the exact order and requested slot to the user, collect approval, and re-check current ownership and order state before writing. If approval expires or the state changes, ask the user again rather than assuming old consent still applies.

🎯 Pro Tip
Build tool authorization tests as if a model could choose any syntactically valid argument. Include wrong owners, invalid enums, oversized inputs, stale approvals, and permission changes.
// Part 04 — Human oversight

Make Approval Specific and Meaningful

Human approval should be reserved for actions whose impact justifies the extra step. Show the proposed operation, affected record, and important consequences in a trusted UI—not only in model-generated prose. Bind consent to the exact operation and arguments, record who approved and when, and invalidate it if the target or amount changes. Do not use a generic "continue" click as approval for a different action.

  1. Classify tools by impact, reversibility, and data sensitivity; default to no write capability.
  2. For approved actions, display the precise target and change using trusted application data.
  3. Bind approval to the specific user, operation, arguments, and expiry; prevent replay.
  4. Re-authorize and validate at commit time, then write an audit event with a correlation ID.
  5. Provide cancellation, rollback, or a support path appropriate to the action.

A human-in-the-loop control only helps when the reviewer has context and meaningful choice. If a workflow presents dozens of vague alerts or hides the action details, approvals can become routine clicks rather than informed checks. Test usability and bypass paths, not just the presence of an approval screen.

// Part 05 — Sandboxing

Contain Code or External Execution

Some agents analyze code, transform files, or run user-provided snippets. Do not execute untrusted code inside the main web application process. If execution is an essential product feature, use an isolated environment with a dedicated low-privilege identity, strict CPU and memory limits, timeouts, restricted filesystem access, and no network access unless the task specifically requires a controlled endpoint. Treat isolation as defense in depth, not proof that arbitrary code is safe.

For ordinary business tools, prefer allowlisted APIs and structured inputs over a general command runner. Keep secrets outside the agent context and sandbox, avoid mounting production credentials, and destroy or reset the environment after each job. Validate outputs before they re-enter trusted systems.

BoundaryControl objective
Process / containerSeparate untrusted work from application and host processes
IdentityUse a non-privileged, task-specific service identity
ResourcesBound runtime, memory, CPU, output, concurrency, and cost
NetworkDeny by default; allow only reviewed destinations where required
FilesystemExpose only task files; prevent access to host or other users’ data
LifecycleReset state, remove temporary credentials, and preserve only approved outputs
💡 Note
Sandbox features depend on the chosen platform and threat model. Consult that platform’s official isolation and networking documentation; do not assume a generic container is a complete boundary by itself.
// Part 06 — Reliability and monitoring

Bound Loops, Fail Closed and Keep Audit Evidence

An agent may repeat tool calls, call a downstream service slowly, or return malformed arguments. Bound total steps, tool calls, elapsed time, retries, output size, and model requests. Give every call a deadline and idempotency strategy appropriate to its operation. If authorization or policy services are unavailable, do not fall back to a broader tool or skip the check.

tool_budget.py
python
from dataclasses import dataclass

@dataclass
class ToolBudget:
    max_calls: int
    calls_used: int = 0

    def consume(self) -> None:
        if self.calls_used >= self.max_calls:
            raise RuntimeError("tool-call budget exhausted")
        self.calls_used += 1

budget = ToolBudget(max_calls=4)
for name in ["search_policy", "get_order_status"]:
    budget.consume()
    print("Allowed bounded call:", name)
print(f"Calls used: {budget.calls_used}/{budget.max_calls}")

Record the actor, tool name, authorization result, approval reference, target identifier, and outcome in protected audit events. Avoid logging credentials, full sensitive arguments, or full prompts unless there is a justified, controlled need. Monitoring should help detect unexpected volume, repeated denials, and unusual sequences without turning logs into another source of data exposure.

// Part 07 — Pitfalls

Common Mistakes

Giving an agent a generic shell or database tool

Why: Broad capability creates a large blast radius and difficult-to-audit behavior.

Fix: Expose narrow business operations with strict schemas and server-side checks.

Trusting model-supplied identity or tenant values

Why: Those fields are untrusted arguments and can differ from the authenticated caller.

Fix: Inject principal context from the verified request and authorize every call.

Treating a confirmation prompt as the permission check

Why: The user may not see the exact effect, and authorization can change before execution.

Fix: Bind approval to the action and repeat authorization at commit time.

Assuming a container makes code execution harmless

Why: Isolation depends on configuration, host, network, secrets, and resource boundaries.

Fix: Use documented isolation controls, deny-by-default access, limits, and platform-specific review.

// Part 08 — Practice

Reduce the Agency of a Delivery Assistant

Practice

Design least-privilege tools and an approval flow

  1. List the assistant’s requested capabilities and divide them into read, propose, and commit operations.
  2. Replace one broad tool with two narrow typed operations and specify server-derived identity and authorization checks.
  3. Design an approval record bound to a specific order and delivery-slot change, including expiry and replay protection.
  4. Write tests for wrong owner, malformed input, changed order state, authorization outage, and exhausted tool budget.
  5. Specify an isolated execution profile for an optional document-conversion task without mounting production secrets.
  6. Define audit fields that support investigation while minimizing personal data and prompt retention.

Hint: Use a fake order and customer. The model can propose a change, but only a verified user approval plus a fresh server authorization can commit it.

🎯 Key Takeaways

  • ✓An agent’s tool call is an untrusted request; the tool service must validate arguments and authorize the actor independently.
  • ✓OWASP LLM06 Excessive Agency concerns excessive functionality, permissions, or autonomy.
  • ✓Prefer narrow, least-privilege tools and separate read, propose, and commit operations.
  • ✓Human approval should display and bind to the exact operation, target, and arguments, followed by authorization at commit time.
  • ✓Sandbox untrusted execution with documented isolation, resource limits, restricted network and filesystem, and no production secrets.
  • ✓Bound agent loops and record privacy-aware audit evidence; fail closed when critical policy checks are unavailable.
  • Reference: OWASP Top 10 for LLM Applications (2025), especially LLM06 Excessive Agency and LLM05 Improper Output Handling.
  • Reference: MITRE ATLAS current knowledge base for AI adversary techniques and mitigations; adapt relevant scenarios to your agent’s actual tools.

Next

What comes next

You have constrained the agent’s actions. Next, protect sensitive information in prompts, logs, and generated outputs.

Lesson 06 · Sensitive Data, Privacy and Output Handling →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...