Dead-letter queues for failing agent jobs

Dead-letter queues isolate terminal failures

A dead-letter queue is a holding queue for agent jobs that have exhausted a defined retry budget. It separates poison work from transient failures so one bad job cannot consume retries forever, while preserving the failed work for inspection and controlled replay.

The important boundary is not the name of the queue. It is the rule that stops normal delivery attempts. A temporary network error, rate limit, or brief provider outage may clear before that budget is exhausted. A job that fails on every attempt under the same conditions should leave the normal queue once it reaches the limit. Continuing to retry it consumes worker capacity without changing the likely outcome.

Cloudflare Queues’ dead-letter queue documentation, updated April 21, 2026, describes the mechanism directly: a message moves after the consumer reaches max_retries, and the dead-letter queue can be consumed independently. If no dead-letter queue is configured, Cloudflare deletes a message that reaches the retry limit. Its documentation also says an unconsumed dead-letter message persists for four days, so a Cloudflare deployment needs an operator or consumer that acts within that window.

A retry limit should cover attempts, not elapsed time alone. An agent job can spend most of its life waiting on tool calls, model responses, or backoff. The surrounding cancellation and timeout budget determines when an individual run must stop; the delivery count determines when the job leaves the normal retry path. Keep those decisions separate in the record so an operator can tell whether the work failed, timed out, or lost its delivery lease.

Dead-lettering does not repair a broken dependency. If a provider is failing across many otherwise valid jobs, moving every job aside merely changes where the backlog accumulates. A circuit breaker around the failing tool can stop new calls while the dependency is unhealthy; the dead-letter queue retains the jobs that already exhausted their attempts.

The record must preserve the failed run

A dead-letter entry is useful only if it carries enough evidence to reproduce the run. The record needs the original payload, the failure history, and a correlation ID. Omitting any of the three leaves an operator guessing about what ran, how it failed, or which telemetry belongs to it.

The original payload is the input the worker actually received, not the current version of a database row from which that input was once assembled. If the queue stores a pointer instead of the body, that pointer must resolve to the same immutable content. A mutable record may have changed by the time someone investigates, turning the replay into a different job while presenting it as the original.

For an agent job, the payload boundary should be explicit. Include every input the worker treats as part of the job contract. Do not assume that a persisted agent checkpoint or conversation record is a substitute: agent memory and state may describe what the agent knew between steps, while the queued payload describes what the worker was asked to start or resume.

Preserving the payload has a concrete cost. It consumes storage and can retain sensitive material longer than the normal processing path. If the queue cannot safely hold the body, preserve an immutable, access-controlled reference plus the version or content hash needed to prove which payload it identifies. A reference to a mutable object does not meet the reproducibility requirement.

The failure history must cover every attempt, not only the final exception. At minimum, retain the attempt number, timestamp, failure reason, and the dependency or agent step that failed. Preserve the error type and available diagnostic detail rather than collapsing every failure into a generic processing error. The sequence matters: three identical validation failures indicate something different from two timeouts followed by an authentication failure, even though both jobs reached the same retry limit.

Azure Service Bus dead-letter queue documentation, updated July 22, 2026, shows why the reason belongs with the message. Service Bus adds dead-letter reason and description properties, uses MaxDeliveryCountExceeded when the delivery limit is crossed, and recommends putting the exception type in the reason and the stack trace in the description for application-level dead-lettering. Its default maximum delivery count is 10, but that vendor default is not evidence that 10 attempts suit a particular agent job. The limit must reflect the job’s retry cost and whether another attempt can encounter meaningfully different conditions.

A complete history also makes the retry policy auditable. An operator can check whether backoff occurred, whether the same dependency failed each time, and whether the final move happened at the configured boundary. Without that sequence, the dead-letter queue records only that processing stopped, not whether the retry system behaved correctly.

Correlation connects the queue record to telemetry

The correlation ID is the stable join key between the original request, queue deliveries, agent run, tool calls, failure logs, and dead-letter record. It must survive handoffs and retries unchanged. A queue-generated message ID may identify one delivery artifact without identifying the wider run, so store the application’s correlation ID explicitly.

Tracing can carry that relationship across services. OpenTelemetry’s trace model, accessed August 26, 2026, defines trace context with a trace ID and span IDs, and explains that context propagation lets spans produced in different places be assembled into a trace. It also provides span links for causally related asynchronous work. For a queued agent job, retain the correlation ID and the relevant trace identifiers in the dead-letter record. If replay starts a new trace, link it to the failed trace instead of making the new execution look like a continuation of the old attempt.

Correlation is especially important when work passes between agents and surrounding systems. A failure may surface in the queue consumer even though the decisive event occurred in a model request, browser action, storage write, or downstream service. The correlation ID lets an operator retrieve that chain without searching by timestamps and hoping concurrent jobs did not overlap.

A correlation ID is necessary evidence, not a replacement for the payload or failure history. It points to telemetry that may be sampled, expired, or unavailable. The dead-letter record therefore needs to remain intelligible when the observability backend cannot return the original trace.

Replay only after the failure condition changes

Replaying a dead-letter job without changing either the payload or the failing dependency usually recreates the same failure. A replay button is not a repair mechanism. It is a new execution of retained work, and it should be used only after an operator can name what is different.

A payload change can correct invalid input, remove an unsupported operation, or update a stale reference. A dependency change can be a restored service, corrected credentials, deployed bug fix, or configuration repair. The operator does not need certainty that the next run will succeed, but there should be a concrete reason the previous failure condition no longer applies.

Preserve the original dead-letter record even when the replay uses an edited payload. The edited job needs its own identity and should record its relationship to the failed job. Replacing the stored payload in place destroys the evidence needed to explain the first run. The same rule applies when the payload is unchanged but a dependency was repaired: record the replay time, the operator or automation that authorized it, and the stated change in conditions.

Bulk replay needs the same gate. A shared dependency recovery can justify replaying a group whose histories identify that dependency. A mixed backlog should not be redriven as one undifferentiated batch merely because the queue has been waiting. Grouping by failure reason and failed dependency keeps the replay decision tied to an actual change.

The queue consumer must also avoid creating a dead-letter loop. If a replayed job exhausts its budget again, its new failure history should point back to the prior dead-letter record while remaining a separate run. That preserves the distinction between repeated delivery attempts and separate, authorized replays.

What to inspect next

After defining the dead-letter record and replay gate, check how the system will reconstruct an agent run after the fact. Confirm the queue’s retry limit, dead-letter retention, payload size and access rules, trace retention, alert ownership, and replay authorization. A dead-letter queue without monitoring and an assigned operator is only delayed data loss.