Backpressure and Admission for Agent Workloads

Backpressure begins before a run is accepted

Backpressure for an agent workload is the control that prevents producers from creating work faster than the system can finish it within its useful lifetime. Admission is the point where the system either commits capacity to a run or declines it; downstream throttles are additional controls, not substitutes for that decision.

Cloudflare’s consumer-concurrency guidance (checked 26 August 2026) documents the operational consequence: when a fixed concurrency limit protects an upstream system, backlog and overall latency can grow. For an agent service, the producer might be an API client, scheduler, webhook, user interface, or another agent. The consumer is the complete execution path, including planning, model calls, tools, state writes, retries, and finalization.

The admission unit should therefore be an agent run, not an individual model request. A run may wait before its first model call, pause between tool calls, retry a failed operation, or retain state while a dependency is unavailable. Every accepted run occupies something even when it is not using model capacity: a queue record, an in-memory object, a workflow slot, a deadline, stored context, or operational attention.

Backpressure starts at admission. Accepting unlimited agent runs and throttling only model calls moves the overload into memory and queue age. The model-call limiter can remain perfectly within its configured rate while the number of waiting runs and the time they spend waiting continue to rise. The service has not controlled demand; it has changed where demand accumulates.

A useful admission boundary covers both queued and running work. A limit on running executions protects workers and dependencies. A separate bound on queued executions limits how much future work the service has promised. Either limit on its own leaves an unbounded place for overload to move into.

Accepted work is a capacity commitment

An accepted run should have a durable place in a bounded queue and a reasonable path to execution before its deadline. Admission does not guarantee success, but it should mean more than recording an intention that may wait indefinitely.

The capacity decision has to reflect the constrained part of the whole run. Model-call concurrency may be one constraint, but agents can also wait on browsers, databases, sandboxes, repositories, approval gates, and third-party APIs. A run that is blocked on a tool still counts as active work if it retains memory, a lease, a browser session, or a worker slot.

Agent runs also differ in size. A queue containing brief classification runs is not equivalent to a queue of repository-wide investigations with many tool calls. A count remains useful as a hard safety bound, but admission can also classify work into known execution classes and reserve separate capacity for them. Any cost estimate used for admission is only an estimate; the hard queue and concurrency bounds remain necessary when actual runs take longer than expected.

The AWS Builders’ Library treatment of queue backlogs (retrieved 26 August 2026) gives a concrete recovery example. If excess traffic is accepted for 30 minutes and the resulting backlog is ten times the consumer’s capacity, draining it takes 300 minutes at that capacity. The arithmetic matters because a short producer-side spike can create a much longer period of poor service after the spike ends.

That recovery debt is especially awkward for agents. Old runs may continue making model and tool calls after the person or upstream workflow that requested them has moved on. Processing the backlog can consume the same capacity needed by current work, so apparent reliability at admission becomes late or irrelevant completion later.

Admission should occur before expensive setup. Authentication, authorization, basic validation, deduplication, deadline checks, and capacity checks can happen without allocating a sandbox or assembling a large prompt. The applicable agent tool permission model still governs what an admitted run may do; admission only decides whether the service can take responsibility for starting it.

Queue depth and queue age describe different facts

Queue depth is the amount of waiting work. Queue age is how long that work has waited. Both are required because neither can stand in for the other.

Queue depth without queue age can hide starvation because a steady depth may contain work that has waited far too long. A queue can finish work at roughly the same rate that new work arrives, leaving its depth visually flat, while a priority rule repeatedly lets newer or higher-ranked runs pass an older run. The depth graph looks stable even though that run is making no progress.

Age should be measured from the original admission timestamp, not from the most recent retry or requeue. Resetting the timestamp on every attempt makes old work appear new and conceals the time already spent waiting. Attempt age is still useful, but it is a separate measurement from total run age.

The most direct queue-age signal is the age of the oldest eligible run. It answers whether any admitted work has exceeded the wait the service intends to provide. Age by workload class or tenant is also needed when a shared aggregate can hide one starved group behind healthy traffic from others.

Completion latency should retain its components:

• time from admission to first execution;
• time spent actively executing;
• time waiting between attempts or for dependencies;
• total time from admission to a terminal result.

These measurements prevent worker time from being mistaken for user-visible latency. A run that executes quickly after waiting a long time still completed late.

Retries make the distinction more important. A repeatedly failing run may consume attempts while making no useful progress. Dead-letter counts identify work that exhausted its retry policy, but they arrive after those attempts. Queue age can show the accumulating delay before the run reaches that terminal state.

Queue depth should still be monitored. Rising depth shows that admitted arrival is exceeding completion. Depth near its bound shows that overflow behavior will soon apply. The operational view should place depth and oldest-work age together so that a flat count cannot be read as proof of health.

A bounded queue requires an overflow policy

A queue bound defines the point at which the system stops accumulating commitments. It does not define what happens next. A bounded queue therefore needs an explicit overflow policy such as reject, shed, or replace; silently growing is not a policy.

The policy belongs in the service contract and in telemetry. The caller, operator, and eventual run record should agree about whether the work was accepted. Returning an accepted response and then discarding the run without a terminal status hides overload as lost work.

Reject

Rejection refuses the new run because no admissible capacity remains. It preserves the work already accepted and gives the producer a clear opportunity to wait, reduce demand, or choose another path.

For an HTTP admission endpoint, RFC 6585’s 429 Too Many Requests status was published in April 2012 specifically for rate limiting. The response should explain the limiting condition and may include Retry-After; the RFC does not prescribe how a server identifies a caller or counts requests. It also states that a 429 response must not be cached.

A retry hint should describe when admission might be attempted again, not promise that capacity will exist then. Clients that retry should avoid synchronized immediate retries, because rejected demand that returns all at once recreates the same admission pressure.

Rejection is the wrong result after the service has told the caller that a run was accepted. Once accepted, cancellation, expiry, or failure should be represented as an explicit terminal outcome. It is also insufficient by itself when one producer can repeatedly consume all newly available slots before other producers are considered.

Shed

Shedding removes work that the service has classified as expendable under overload. The classification must be established before pressure occurs: expired work, duplicate work, best-effort refreshes, and work whose result no longer has a consumer are possible categories only when the product contract actually permits their removal.

Shedding creates capacity by declining or terminating work rather than waiting for every item to run. Its cost is a known loss of completion. That loss must be visible through a rejection or terminal status and a metric carrying the reason.

Shedding is wrong for work whose side effect must occur exactly as requested, or where the service cannot establish that the work has become valueless. It must not be inferred merely from queue age. An old run can be urgent, and a recent run can already be obsolete.

Deadlines provide a defensible shedding condition when they are part of the request contract. A run that cannot produce a useful result after its deadline should not consume scarce execution capacity merely because it reached the head of the queue. Its terminal state should distinguish expiry from execution failure.

Replace

Replacement admits new work by removing older queued work that the new request supersedes. It requires a stable supersession key and a rule proving that only one pending result remains useful. The queue can then keep the newest desired state instead of executing every intermediate request.

Replacement is useful only where work represents a replaceable target state. It is wrong when requests are independent events, when each run has a required side effect, or when an older run has already begun an irreversible action. Replacing queued metadata does not undo tool calls already made by a running agent.

The replaced run needs a visible terminal state, and the new run needs its own admission timestamp. Reusing the old timestamp would overstate the new run’s age; deleting the old record would erase evidence that the overflow policy was invoked.

Fair admission prevents one workload from owning the queue

A single global bound protects total capacity but does not guarantee fair access. One tenant, repository, scheduled job, or recursive agent can fill every queue slot and cause unrelated work to be rejected or delayed.

Fairness therefore has to be applied while admitting work. Per-tenant or per-workload limits can sit beneath the global bound, leaving capacity available for other producers. Separate queues can provide stronger isolation where workloads have different deadlines or consumers, though they add scheduling and operating work.

Priority affects order after admission; it does not create capacity. A priority queue can keep urgent runs responsive while allowing low-priority runs to age indefinitely. Age by priority class and a stated aging or expiry rule are needed if every accepted class is promised eventual service.

A shared queue also makes aggregate backpressure imprecise. Rejecting every producer because one workload caused the backlog penalizes healthy workloads. Conversely, measuring only total depth can conceal that one workload has reached an unacceptable age. Admission metrics need the same workload dimensions used by the policy.

Recursive and fan-out behavior must return through the same boundary. An admitted parent run should not gain permission to enqueue unlimited child runs. Child work either consumes capacity reserved for the parent or passes admission independently. Otherwise, the external queue appears bounded while the unbounded queue has merely moved inside the agent.

The amount of fan-out is also shaped by how much work the agent decomposes before acting. A plan that creates many runnable children at once has a different admission footprint from one that releases the next child only after the previous result is known.

Retries remain subject to admission capacity

A retry is additional execution demand, even when it belongs to an already accepted run. Unlimited retries can monopolize worker slots and delay first attempts for new work.

The retry policy should bound attempts, retain the original run age, and stop when the result can no longer arrive before its deadline. Delayed retries need a bounded place to wait. Moving them to a separate retry queue can isolate first attempts, but that queue still needs capacity, age measurements, and overflow behavior.

Retry eligibility is different from retry readiness. A transient failure may allow another attempt, while current pressure may require that attempt to wait. Immediate retry loops bypass queue admission and convert dependency trouble into local concurrency and memory pressure.

A dead-letter queue is a terminal holding area for work that exhausted delivery or processing attempts, not extra invisible capacity for the active queue. Its growth should be explicit, and replay should pass through admission again. Bulk replay without a capacity check can recreate the original overload.

Admission telemetry must account for every decision

The admission boundary should count accepted, rejected, shed, replaced, expired, cancelled, completed, and failed runs. Every non-accepted decision needs a reason such as global capacity, tenant capacity, duplicate, superseded work, or expired deadline.

Queue telemetry should include current depth, the configured bound, oldest eligible age, admission rate, start rate, completion rate, and retry traffic. These are measurements, not universal thresholds. The correct limits come from the service’s tested capacity and the maximum wait its callers can still use; no benchmark for agent workloads has been supplied here.

Running-work telemetry should cover active runs and constrained resources such as model-call slots, tool slots, sandboxes, and browser sessions. This shows whether a full queue is waiting on actual execution capacity or on a policy that has stopped selecting some work.

The run record should preserve admission time, first-start time, attempt timestamps, terminal time, workload class, tenant, priority, deadline, and overflow outcome. That record supports both user-visible status and later policy review. The rules for retaining it belong with the broader design for agent memory and state.

Alerts should be tied to the promise made at admission. Queue depth approaching its bound warns that overflow decisions are imminent. Oldest-work age approaching the useful deadline warns that accepted work is becoming stale. A low depth does not cancel an age alert, and a low age does not make an unbounded queue acceptable.

The admission contract should be testable

The policy is complete when each request receives one traceable result: accepted into a bounded class, rejected before acceptance, shed under a named rule, or used to replace specifically identified queued work. No path should create an unbounded in-memory wait list beside the queue.

Load tests should exercise the bound and confirm the documented overflow result. They should also hold depth steady while varying priorities to verify that oldest-work age exposes starvation. Dependency slowdown tests should confirm that model and tool throttles do not allow accepted runs, retained state, or retries to grow without limit.

Capacity changes should update the admission bound deliberately. Adding model throughput does not automatically add browser, sandbox, database, or tool capacity. Likewise, moving the agent among the available deployment topologies changes where work waits but does not remove the need for a bounded admission point.

The next values to establish are the maximum useful queue age for each workload class, the tested running concurrency for every constrained dependency, the global and per-tenant queue bounds, and the precise response for each overflow outcome. Teams should then connect those limits to the agent’s spend controls, because admitted work can remain within a concurrency limit while exceeding its intended spend ceiling.