Trace Sampling: Keeping the Traces That Matter When You Can't Keep Them All

Distributed tracing's pitch — every request's journey across services, reconstructible on demand — collides at production scale with arithmetic nobody escapes: a mid-size estate serving 50k requests/sec, each touching a dozen services with a handful of spans apiece, generates billions of spans daily, and storing them all costs more than the incidents they'd explain. Tracing at scale is therefore a retention-policy problem: which traces do you keep, decided by what logic, enforced where — and the quality of those decisions determines whether the tracing system answers the 3 AM question ("show me a trace of the failure") or delivers its most demoralizing failure mode: perfect instrumentation, 1% sampling, and no trace of the incident anyone is debugging, because uniform randomness keeps mostly the healthy majority.

#The consistency rule that everything rests on

Before strategy: a partial trace is a lie — spans sampled independently per service produce journeys with missing legs that mislead worse than no trace. The foundational mechanism is consistent, propagated sampling decisions: the decision made once (typically at the ingress span) and carried in the trace context (the W3C traceparent's sampled flag — the same propagation rails as every context discipline), so every downstream service honors the same verdict and traces arrive whole or not at all. Every strategy below is a policy about where and on what information that decision gets made — which is the entire design space.

#Head vs tail: deciding early and cheap, or late and smart

Head sampling decides at trace start with what's knowable then — essentially nothing about how the trace will turn out — so its policies are rate-shaped: probabilistic percentages, rate limits per service, per-route overrides (checkout at 100%, health checks at 0% — the criticality tiering from every SLO discipline). Its virtues: trivially cheap (unsampled traces cost near-zero — the spans are never exported, or never created), simple, and predictable in cost. Its structural flaw: it cannot know the future — the error, the 30-second outlier, the interesting anomaly all look identical to the boring median at decision time, so head sampling keeps them at the same rate as everything else, which at 1% means missing 99% of exactly what you wanted.

Tail sampling inverts the timing: buffer every trace's spans (in a collector tier), decide after completion with full knowledge — keep all errors, keep everything over the latency threshold, keep traces touching flagged tenants or routes, downsample the healthy fast majority to a token rate. This is the policy that matches what humans actually query for ("show me slow," "show me failed"), and it's what the OpenTelemetry collector's tail-sampling machinery and the vendor pipelines implement. The bill: the buffering tier is real infrastructure (every span held in memory for the decision window — the capacity math scales with trace volume and duration, and long-running traces strain the window), decisions require all the spans to converge on one collector (trace-ID-based routing in the collector fleet — the consistent-hashing pattern, again), and cost predictability degrades (an incident producing 100% errors is also producing 100% keep-decisions at your worst moment — the sampling policy needs its own rate cap, per the protect-the-protector instinct).

The mature architecture is layered, as usual: head sampling as the volume governor (drop the health checks and the known-noise at the source — the cheapest span is the one never created), tail sampling as the intelligence layer (error/latency/attribute-biased retention on what survives), and — the piece that makes relaxed sampling tolerable — metrics and exemplars carrying the aggregate truth: the span-derived RED metrics are computed before sampling (so dashboards and SLO math see 100% of reality regardless of trace retention — the histogram-sketch machinery from observability practice), with exemplars linking metric buckets to retained trace IDs (the "click the p99 bucket, land on a real slow trace" workflow that makes sampled tracing feel unsampled exactly where investigation starts).

#The operational rim

The details that separate working pipelines from aspirational ones: sample-rate metadata on every kept trace (the decision and its rate recorded as span attributes, so analytical queries can re-weight — a kept-at-1% trace represents a hundred; count-style analysis over traces without rate-awareness is silently wrong arithmetic), per-signal budgets with owners (trace volume by service on the same FinOps-style dashboards as log ingestion — the cardinality-and-cost review from logging practice, applied to spans; the service exporting 40% of the estate's spans for no queried benefit is this pipeline's unused-index equivalent), dynamic and per-tenant overrides (the debug-sampling toggle — crank a route/tenant to 100% during investigation without a deploy, per the runtime-toggle disciplines; and the whale tenant's traces rate-capped per the fairness rules), and retention tiers matching query patterns (hot full-fidelity days for incident work, downsampled/aggregated weeks for trends — the lifecycle economics, as everywhere). And the honesty check that keeps the whole system accountable: periodically, during a real incident review, ask "did we have the trace?" — the tracing pipeline's single meaningful SLO is trace-availability-for-investigated-incidents, and a quarter of misses is the signal to rebalance policies toward the tail.

The closing frame: sampling isn't the compromise tracing makes with cost — it's the design surface where an observability team encodes what questions the organization has committed to answering. Keep the errors, keep the outliers, keep whole traces or none, let metrics carry the aggregates, link them with exemplars, and cap the volume everywhere else — and 1% storage delivers 95% of the answers, which is the trade every honest telemetry budget was always going to make. The only unacceptable policy is the accidental one.