Running Agents Under Provider Rate Limits

Before you start

You need three things in place before any of the steps below will hold.

The provider’s published limits for the key you are actually using. Not the tier above, not the marketing page. Anthropic publishes its per-model limits in the API rate limits reference, and the numbers differ by model and usage tier. Write yours down; every threshold below is derived from them.

A single choke point for outbound model calls. If three parts of your agent construct their own HTTP client, you cannot bound concurrency at all. One client, one place where requests are admitted.

Per-request token accounting. You need the input and output token counts from each response, or at minimum an estimate before the call. Without them you can only govern one of the two axes, and you will be blind on the one that actually blocks you. If you already emit per-call cost data, the same instrumentation covers this — see attributing agent cost and latency to the work that caused it.

Steps

1. Work out your real request rate, then divide the limit by your fan-out.

   An agent that fans out to N parallel tasks multiplies its own request rate by N, so the provider rate limit, not the machine, is the first ceiling it hits. A single-threaded loop making one call every two seconds sits at 30 requests per minute. Fan that same loop out to 12 subtasks and it is asking for 360 — the laptop is idle and the API is returning errors. Nothing about the machine changed; the arrival rate at the provider did.

   The mechanism is that rate limits are enforced at the account or key, not the process. Horizontal scaling gives you no relief unless you also spread across keys, which most providers’ terms treat as circumvention. Do this arithmetic before writing the pool, because it gives you the number the pool needs.

2. Measure both axes separately.

   Rate limits are usually enforced on two independent axes at once — requests per minute and tokens per minute — and a workload can be well under one while being blocked by the other. Two shapes of agent fail in opposite directions. A classifier firing hundreds of 200-token prompts saturates requests per minute with trivial token spend. A single agent stuffing 150k tokens of repository context into each call burns the token budget in a handful of requests while the request counter barely moves.

   Log both counters per minute and look at the ratio. Whichever fills first is the one you tune against; optimising the other is wasted engineering time. If the token axis is your binding constraint, the fix is often not concurrency at all — it is sending less context, or sending the cheap calls to a smaller model, which is the routing decision covered in routing requests between models and in the frontier versus mid-tier comparison.

3. Put a bounded worker pool in front of every call.

   A bounded worker pool with a queue makes the concurrency ceiling an explicit number you chose, rather than an emergent property of how fast the agent happens to run. Without one, your effective concurrency is whatever falls out of task duration, retry timing and how many subtasks the planner decided to spawn this time — a number nobody picked and nobody can reason about.

   Concretely: a fixed set of W workers pulling from a queue, and every model call goes through it. Set W from step 1 — limit divided by fan-out, with headroom. The cost is real: work now waits in the queue, so end-to-end latency for the last task rises, and you need a queue depth bound or a burst of subtasks becomes unbounded memory. It is also the wrong tool when the token axis is your constraint and request sizes vary wildly — four workers each sending 100k tokens will breach a token limit that forty workers sending 1k tokens would not. In that case gate on a token budget rather than a worker count.

   Whatever number you pick, write it as one named constant. A concurrency ceiling scattered across three call sites is a ceiling you will violate during the next refactor.

4. Honour Retry-After instead of guessing.

   HTTP 429 responses carry a Retry-After header, and honouring it is cheaper and more reliable than a fixed backoff guess. RFC 6585, which defines 429 Too Many Requests, specifies that the response may include Retry-After indicating how long to wait. The value is authoritative in a way your guess cannot be: the server knows when its window resets, and you are inferring it.

   A fixed guess fails in both directions. Sleep 30 seconds when the window resets in 2 and you have thrown away 28 seconds per worker. Sleep 2 when it resets in 30 and you spend the interval generating fifteen more 429s, each one a round trip that costs you and the provider something and moves nothing forward.

   Parse the header, sleep for that duration, then retry. Fall back to exponential backoff only when the header is absent — some responses omit it, and a client that assumes it is always present will crash on the one that isn’t.

5. Add jitter to every retry delay.

   Retrying a rate-limited request without jitter synchronises every worker onto the same retry instant and reproduces the burst that caused the limit. Twenty workers rejected in the same second, all sleeping exactly 5 seconds, all wake in the same millisecond and fire twenty simultaneous requests. That is the original burst, delayed. The limit rejects them again and the cycle repeats.

   Randomising the delay spreads those retries across the window. AWS’s analysis of timeouts, retries and backoff with jitter walks through why: without jitter, backoff synchronises clients into clusters, and adding randomness both cuts contention and reduces total work done. Sleeping a random duration up to the computed backoff — rather than the backoff itself — is the standard form.

   This applies to the Retry-After path too. If the header says 12 seconds and every worker sleeps exactly 12, they resynchronise on the reset instant. Sleep the header value plus a small random offset.

6. Cap retries and surface the failure.

   Retry forever and a sustained limit turns into an agent that appears to be working while making no progress. Pick a maximum attempt count, and when a task exhausts it, fail it loudly with the last Retry-After value attached — that value tells whoever reads the log whether the fix is more patience or a higher tier. Long-running agents that must survive a multi-minute limit window are better served by checkpointing to durable execution than by a longer retry loop holding a process open.

What done looks like

The agent has one bounded worker pool with a concurrency limit set from the published rate limit divided by observed fan-out, expressed as a named constant. Both requests-per-minute and tokens-per-minute are logged per minute, so you can name which axis binds. Every 429 is handled by reading Retry-After, sleeping that duration plus a random offset, and retrying up to a fixed cap; a missing header falls back to jittered exponential backoff. Under sustained load the request rate flattens against the ceiling instead of oscillating between bursts and rejections, and a limit that outlasts the retry cap produces a failed task with the wait time in the log rather than a silent stall.