Trace Sampling for High-Volume Agent Systems

What trace sampling controls

Trace sampling limits which distributed traces are processed and exported when recording every agent run would produce more telemetry than the system can economically retain. The decision should apply coherently to the whole trace: keeping unrelated individual spans reduces volume, but it does not preserve the path of one run through its services.

OpenTelemetry’s sampling guide, last modified October 16, 2025, says sampling is worth considering at 1,000 or more traces per second, particularly when most traffic is healthy and has little variation. That is a condition for considering sampling, not a universal threshold. Low-volume systems may gain little from it, while systems that are required to retain all telemetry should not discard traces at all.

Sampling saves storage, export bandwidth, and backend processing only by giving something up: some events will no longer be available as complete traces. The policy therefore has to reflect what the traces are used for. A representative probability sample can describe ordinary traffic, while an outcome-based sample can preserve unusual runs for diagnosis. Those two datasets answer different questions.

Head sampling decides before the result exists

Head sampling makes the keep-or-drop decision when a trace begins, before the system knows its final latency, error state, or cost. The sampler can use information available at that point and can apply a deterministic probability based on the trace identifier, but it cannot inspect an outcome that has not happened.

That timing makes head sampling efficient. The decision can be propagated immediately, and services do not need to buffer a completed trace before deciding whether to export it. A deterministic trace-level decision also allows all participating services to keep or drop the same trace without coordinating again at the end.

The cost is selection blindness. Head sampling can miss rare expensive failures because neither the failure nor its expense is known when the decision is made. With a uniform 1% head-sampling rate, any individual failure has roughly a one-in-100 chance of being retained, assuming no special rule applies and trace identifiers are evenly distributed. A failure class that occurs only a few times may consequently leave no retained example even though the sample represents routine traffic adequately.

Head sampling fits when bounded telemetry volume and low sampling overhead matter more than retaining every unusual outcome. It is the wrong sole mechanism when the explicit requirement is to keep all traces that exceed a latency or cost threshold, or all traces that finish with an error. Increasing the probability improves the chance of catching those traces, but it does not turn an early decision into an outcome-aware one.

For agent systems, that distinction affects later analysis. A probability sample can support a view of common run paths, but it cannot guarantee evidence for the particular model call or tool invocation that made one run costly. That investigation also depends on attributing cost and latency to the work that caused it before the trace is evaluated or exported.

Tail sampling uses the completed outcome

Tail sampling delays the decision until the sampler has received all or most spans in a trace. According to OpenTelemetry’s tail-sampling documentation, the retained set can be selected using errors, overall latency, or attributes found on spans. If cost is recorded in the trace before the decision, cost can be another retention condition.

This timing is what allows a policy to keep a failed run even when failures are rare, retain a slow run even when its individual spans looked ordinary at the start, or preserve a costly run after its usage has been totalled. The policy operates on the result rather than trying to predict it.

The mechanism requires buffering complete traces. Spans arrive over time, so the tail sampler must hold trace state until it decides that enough of the trace has arrived to evaluate the policy. That consumes memory or other temporary storage, delays export, and adds a stateful component that must keep up with peak trace volume. OpenTelemetry notes that large systems may need many compute nodes for this work and that the sampler may need a simpler fallback if it cannot keep pace.

Completeness also needs an operational definition. Ending the collection window too early can exclude a late span; waiting longer increases buffer occupancy and export delay. A system with long-running or asynchronous agent work therefore needs a trace completion rule and enough capacity for the resulting decision window. If the buffer overflows, the sampling policy is no longer being applied to the complete population it was designed to inspect.

Cost-based retention has another condition: the relevant cost must exist as a span or trace attribute before the tail decision. A cost added only after export cannot influence that decision. The same applies to error classification or a latency total calculated outside the sampling pipeline.

Tail sampling is wrong when the buffering and stateful processing cost exceeds the value of outcome-aware traces, or when telemetry must be exported with minimal delay. It also cannot recover spans already discarded by an earlier head sampler. Combining the two mechanisms protects an overloaded collection pipeline, but any trace rejected at the head is unavailable to later tail rules, including rules intended to preserve rare expensive failures.

One trace needs one consistent decision

A sampling policy fails structurally when services make incompatible decisions. One service can record the root and its local spans while a downstream service drops its part of the same trace. The retained result then looks complete at the entry point but has a hole where the downstream work should be, weakening both diagnosis and agent-run replay.

The W3C Trace Context Recommendation, published November 23, 2021, defines the sampled flag in the traceparent header for communicating recording behavior between services. A component that makes a definitive recording decision should reflect it in that flag, and a component making its own decision should respect the received value. For deferred or delayed decisions, the recommendation says to propagate the flag unchanged.

The flag is a coordination signal, not proof that every span was successfully stored. A set value means the caller may have recorded trace data. Export failure, instrumentation gaps, or a missing propagation path can still produce an incomplete trace. Consistency therefore covers both the decision and the transport of trace context across every service boundary involved in the run.

A downstream service must not silently replace a propagated trace-level decision with an unrelated local probability. If services need different volume controls, those controls still have to preserve coherent traces. Deterministic sampling based on the same trace identifier is one way to keep an early decision aligned; parent-based behavior that follows the incoming decision is another expression of the same requirement.

Tail sampling adds a second consistency boundary. Candidate spans must reach the component responsible for the delayed decision instead of being dropped by downstream services first. When several tail-sampling instances share the load, the deployment must ensure that the spans belonging to one trace are evaluated as one trace; otherwise each instance can see only a fragment and apply rules to incomplete evidence.

External callers should not receive unlimited authority to force expensive recording merely by setting the sampled flag. The W3C specification identifies denial-of-service and tracing-cost risks from blindly honoring untrusted trace context. A trust-boundary policy may validate, rate-limit, or restart incoming context, but after that boundary establishes an internal decision, the services behind it still need to propagate that decision consistently.

What to check next

Record the sampling policy alongside its rate, decision point, retention conditions, buffer limits, and overflow behavior. Then verify with a multi-service test that both retained and rejected traces carry the same decision through every hop, and that a retained trace includes the expected downstream spans.

Check that error, latency, and cost fields are populated before a tail decision; that long-running traces fit within the completion window; and that sampler saturation is observable. Sampling governs which evidence survives, while trace-data retention boundaries govern how long that surviving evidence remains available. If identifiers must join traces to model calls or tool activity outside the tracing backend, define correlation identifiers across agent runs as a separate contract rather than expecting sampling to provide that linkage.