Isolation Boundaries for Multi-Tenant Agent Runtimes

Isolation boundaries for multi-tenant agent runtimes

An isolation boundary is a place where an agent runtime prevents one tenant’s identity, data, authority, or resource use from flowing into another tenant’s work. The boundary must survive background execution, retries, caching, and tool calls—not merely the first HTTP request.

Tenant identity belongs in every storage key

Tenant identity must be part of every storage key and tool authorization check, not only the incoming HTTP route. A route such as /tenants/acme/runs/123 establishes request context, but it does not constrain later database queries, object-store reads, queue messages, or tool calls by itself.

A useful key shape makes the tenant namespace explicit:

tenant:{tenant_id}:agent:{agent_id}:run:{run_id}:artifact:{artifact_id}

The order matters less than the invariant: a globally unique run or artifact identifier must not become a substitute for tenant scope. Random identifiers reduce guessing; they do not prove that the caller may access the object. A lookup should therefore require both the tenant identity and the object identity, even when the object identifier is difficult to predict.

Tenant context should come from authenticated, trusted runtime state. Do not let a prompt, model response, tool argument, or user-editable metadata field choose the effective tenant. Those values may describe a target, but the authorization layer must compare that target with the tenant already bound to the run.

This requirement reaches every persistent form of agent state:

• conversation and run records;
• checkpoints and resumable workflow state;
• tool inputs, outputs, and approval records;
• uploaded files and generated assets;
• embeddings, indexes, and retrieval documents;
• scheduled work, retries, and dead-letter records;
• traces, evaluations, and billing or usage records.

Including tenant_id in a row is not sufficient if queries can omit it. Composite primary keys, tenant-scoped repositories, database policies, or separate stores can make omissions harder, but the chosen mechanism must prevent an unscoped lookup from becoming the normal path. The cost is extra schema, migration, and test work. The benefit is that tenant scope becomes a property of data access rather than a convention remembered by each caller.

Authorization must be repeated at the point of use

The OWASP Authorization Cheat Sheet, accessed 2026-08-26, recommends least privilege, denial by default, permission validation on every request, object-specific checks, server-side enforcement, logging, and authorization tests. In an agent runtime, “every request” includes internal requests that occur after the public route has returned.

A multi-step run may call a tool minutes later, resume from a queue, or retry after credentials and permissions have changed. The tool authorization decision must use the current tenant, the agent or user identity, the requested action, and the target resource. A check performed when the run began cannot authorize every future action unconditionally.

The public gateway is one enforcement point, not the whole enforcement system. The worker that reads storage, the service that invokes a tool, and the tool adapter that performs the side effect each need enough trusted context to reject a mismatch. If a downstream service receives only document_id=123, it cannot distinguish an authorized call from a cross-tenant reference. It needs the tenant scope and the identity whose authority is being exercised.

This is also why an agent should not silently borrow a broad service credential. Give the runtime a defined principal and preserve the tenant in that principal’s authorization context. The related design question is covered in giving an agent its own identity, while the action-level rules belong in agent tool permission models.

Authorization failure must stop the operation before any read or side effect. Logging the mismatch is a detective control, not permission to continue. Error handling should avoid returning another tenant’s identifiers, object metadata, cached content, or tool output.

Shared caches need authorization context in their keys

Shared caches can cross tenant boundaries when the cache key omits authorization context. A cache keyed only by prompt hash, URL, document identifier, model name, or tool arguments can return a value created under another tenant’s permissions.

The mechanism is simple: the cache treats two requests as equivalent even though the authorization system does not. Once the first tenant’s result is stored, the second tenant receives that stored result without repeating the protected read or tool call. Correct authorization in the underlying database does not help if a cache hit bypasses it.

This applies to more than HTTP response caches. Agent runtimes commonly reuse intermediate values such as retrieval results, document summaries, tool responses, transformed assets, policy decisions, and model outputs. Any reused value derived from tenant data or tenant-specific permissions is authorization-sensitive.

At minimum, its key needs tenant identity. It also needs every authorization attribute that can change whether the value is valid for the caller. Depending on the application, that can include the principal, role or relationship, resource version, policy version, and data classification. The key should contain stable identifiers or hashes rather than raw credentials or sensitive attributes.

A tenant-scoped key prevents direct reuse across tenants, but invalidation still matters within one tenant. A cached result created before access was revoked can remain too permissive. Short expiration, explicit invalidation, resource-version keys, or bypassing the cache for sensitive operations can limit that window. Which control is appropriate depends on how quickly permission changes must take effect.

Caching authorization decisions deserves the same treatment as caching data. A result such as allow:read:document-123 is incomplete unless it is bound to the tenant, principal, action, resource, relevant policy state, and expiration. “Allowed once” is not a durable property of the document.

The engineering cost is lower cache reuse and more complicated invalidation. That is the correct trade when two requests are not actually equivalent. A cache should be shared across tenants only for content proven to be public and independent of tenant configuration, credentials, or data.

State partitions must use tenant-scoped identity

A stateful runtime can assign each tenant its own coordinator or state partition. Cloudflare’s Durable Objects documentation, accessed 2026-08-26, describes globally named objects that combine compute with private, transactional, strongly consistent storage attached to each object.

That primitive can form a useful tenant state boundary only when routing chooses an object identity that includes the tenant. Naming an object solely from run_id, room_id, or user_id is unsafe if those identifiers can overlap across tenants or if one user can belong to several tenants. A name derived from both tenant and local resource identity keeps the routing decision aligned with the storage boundary.

The object boundary does not eliminate authorization checks. A caller may still request the wrong tenant’s object, and deterministic names are routing inputs rather than credentials. The entry point must derive or validate the object name from trusted tenant context, and methods inside the object must reject conflicting tenant claims.

A per-tenant object also changes the failure domain. State and serialized work for that tenant can be coordinated without placing every tenant in one shared state machine. It does not, by itself, establish that all downstream storage, caches, logs, model calls, or external tools follow the same partition. Each of those boundaries needs its own tenant scope.

Process and storage isolation protect different paths

Logical tenant checks constrain what correct application code may access. Process isolation constrains what code in one execution environment can directly read from another environment’s memory or local files. Storage isolation constrains which persistent data a runtime or credential can reach.

These layers matter because an agent run handles data outside the primary database. Prompts, retrieved passages, tool results, generated files, credentials, and approval state may live in memory or temporary storage while the run is active. Reusing a worker process can leave pools, globals, temporary paths, or in-memory caches available to the next run unless the runtime clears them or separates execution contexts.

Storage isolation can use tenant-scoped keys and enforced query predicates, separate schemas or databases, separate object-store prefixes with scoped credentials, or a stronger physical separation required by the workload. The label “separate” is not enough: the credential presented by the runtime determines what the boundary actually enforces. A tenant-specific prefix protected by a credential that can read the entire bucket remains dependent on application checks.

Process isolation has a real operating cost. More execution contexts increase startup work, scheduling complexity, observability needs, and capacity overhead. Storage isolation adds provisioning, migrations, backup handling, and cross-tenant operational work. Those costs should be accepted where the data sensitivity or failure impact requires the stronger boundary, rather than hidden behind a claim that tenant IDs alone create isolation.

NIST SP 800-53 Rev. 5’s security and privacy control catalog, accessed 2026-08-26, provides a customizable set of controls for managing system and organizational risk. For runtime design, the useful principle is defense in depth: access enforcement, information-flow restrictions, audit, resource controls, and system protection remain distinct responsibilities. Passing one authorization check does not satisfy all of them.

Per-tenant quotas contain resource contention

Per-tenant quotas limit noisy-neighbor effects but do not replace process or storage isolation for sensitive data. A quota controls how much of a resource a tenant may consume; it does not control which tenant’s data a request can read.

Useful quota dimensions follow the scarce resources in the runtime:

• concurrent agent runs and tool calls;
• queued and retrying work;
• model requests or tokens;
• execution time;
• retrieval and storage operations;
• generated files and retained run data;
• outbound requests to third-party tools.

The quota key must include tenant identity for the same reason storage and cache keys do. A global counter can protect the whole service but cannot stop one tenant from consuming the shared allowance. A counter keyed only by user can also fail when users belong to multiple tenants or automation runs without a human user.

Quota enforcement should happen before committing the scarce resource. Counting only completed work leaves failed, slow, or repeatedly retried operations outside the limit even though they consume capacity. Retries need a defined accounting rule so that changing a message identifier does not reset the tenant’s usage unintentionally.

A tenant quota limits the reach of overload: excess work can be delayed or rejected without allowing that tenant to occupy every worker or queue slot. It cannot prevent a missing tenant predicate, a cache-key collision, a shared credential from reading the wrong store, or residual data in a reused process. Treating quotas as a confidentiality control leaves the actual disclosure paths untouched.

Quotas also need a service-wide ceiling. Per-tenant limits multiplied across many active tenants can still exceed total capacity. The tenant limit answers who may consume a share; the global limit protects the runtime as a whole. Neither grants access to data.

Queues, schedules, and retries must carry the boundary forward

Background execution is where route-only isolation usually disappears. A queue message should carry an immutable tenant identifier together with the run, action, and target identifiers. The consumer must validate that the referenced records belong to that tenant before loading state or calling a tool.

Scheduled jobs need the same context because there may be no live user session when they run. A scheduler record should bind the tenant and authorized operation at creation. At execution time, the runtime should re-evaluate permissions that can change rather than assuming the old decision remains valid.

Retries must not reconstruct tenant identity from an object discovered during the failed operation. They should resume with the original trusted tenant context and an idempotency key scoped to that tenant. Otherwise identical retry identifiers from different tenants can collide, or a partially completed operation can be resumed under the wrong scope.

Logs and traces are tenant data too

Observability systems often collect the most revealing parts of an agent run: prompts, retrieved text, tool arguments, error messages, and returned identifiers. Tenant isolation therefore applies to log ingestion, trace storage, search, export, and deletion—not only to the application database.

Each event should carry tenant identity as structured metadata, alongside the run and span identifiers. A correlation identifier explains which events belong to one execution; it does not establish who may see them. The distinction is developed further in correlation identifiers across agent runs.

Access to observability data must apply the same tenant authorization policy as access to run data. Redaction reduces what is stored, while retention limits how long it remains exposed; neither substitutes for access control. The policy for conversations, traces, and tool results should be explicit in the system’s agent data-retention boundaries.

The boundary must be testable

Isolation claims should be expressed as invariants that automated tests can try to break. Create two tenants with deliberately overlapping local identifiers, then attempt reads, writes, cache hits, retries, exports, and tool calls across them. A test that uses globally unique fixtures can miss an omitted tenant key because the accidental lookup still finds only one record.

Authorization tests should cover denial paths as well as successful access. Remove or change permission after a run begins, resume the run, and verify that the next protected action is denied. Seed a cache under one tenant and request the same logical input under another. Send a queue message whose tenant conflicts with the referenced object. Confirm that failed checks do not expose protected metadata in errors or logs.

Operational checks should verify the enforcement point, not only the schema. A column named tenant_id, a tenant field in a token, or a per-tenant dashboard does not prove that storage queries, cache keys, tool adapters, and exports use it.

What to check next

Before approving a multi-tenant runtime, identify the trusted source of tenant identity and trace it through HTTP handling, run state, queues, caches, tool adapters, temporary files, logs, and deletion workflows. For each boundary, record what enforces the tenant match, what credential can bypass it, and how a cross-tenant denial is tested.

Then check the decisions that interact with isolation: whether agents have distinct identities, whether every tool action is authorized at execution time, whether retention applies to derived data as well as conversations, and whether quotas cover both tenant-specific and service-wide capacity. The design is complete only when tenant context remains enforceable after the original request is gone.