Sampling: How Much Traffic Do You Need to Log?

Sampling is usually framed as a cost decision, which makes it sound like a slider between “expensive” and “blind”. It is really a design decision about which questions you intend to be able to answer, and those questions have arithmetic attached.

Three questions sampling has to answer

Before choosing a rate, decide which of these you need, because they have different requirements and only the first is cheap.

• “What is the aggregate behaviour?” Rates, percentiles, spend. Needs a modest sample, or better, unsampled metrics — see below.
• “What happened to this request?” Needs that specific trace to have survived. A uniform 1% sample answers this 1% of the time, which is to say: never, when someone asks.
• “What is this rare failure?” Needs enough examples of an event that is rare by construction. This is the one the arithmetic is about.

The design that satisfies all three is not a single rate. It is unsampled metrics, an unsampled request log with content omitted, and a biased trace sample that keeps everything interesting.

Head sampling and tail sampling

For LLM workloads tail sampling is nearly always the right answer, because the interesting property — the error, the 40-second latency, the 200,000-token prompt — is only known at the end. The usual arrangement is a light head sample to shed obvious volume, then tail policies for everything that matters.

One correctness point that gets missed: if you count sampled spans and present the result as a total, the number is wrong by the sampling factor. OpenTelemetry addresses this with adjusted counts propagated in tracestate, so that a sampled span carries the population it represents. If your backend does not do that for you, do aggregate counting from metrics, not from traces.

The arithmetic of rare events

Suppose an event occurs on a fraction q of requests and you keep a fraction s of traces. Over N requests, the probability of capturing at least one is 1 − (1 − q·s)^N. Solving for 95% confidence gives a rule that is easy to carry around:

N ≥ ln(0.05) / ln(1 − q·s)   ≈   3 / (q · s)

    q = 1 in 1,000    s = 1.00  →  N ≈      3,000 requests
    q = 1 in 1,000    s = 0.01  →  N ≈    300,000 requests
    q = 1 in 100,000  s = 1.00  →  N ≈    300,000 requests
    q = 1 in 100,000  s = 0.01  →  N ≈ 30,000,000 requests

And for characterising rather than merely seeing it — the relative
standard error of a count is 1/sqrt(expected count), so:

    ~100 captured examples  →  ±10% on the rate
    ~400 captured examples  →  ±5%  on the rate
    expected count = N · q · s

Read the third and fourth lines together. At a million requests a day, a one-in-a-hundred-thousand failure produces ten instances daily. Keep 1% of traces and you see one every ten days, which is not investigable. Keep 100% of errored traces and you see all ten, at a storage cost of ten traces. That asymmetry is the entire argument for biased sampling: the traces you want are, by definition, the rare ones, and keeping all of them is cheap precisely because they are rare.

A tail-sampling policy set

The OpenTelemetry Collector’s tail sampling processor evaluates a list of policies and keeps a trace if any of them says keep. A starting set for an LLM service:

processors:
  tail_sampling:
    decision_wait: 15s          # must exceed your slowest expected trace
    num_traces: 100000          # traces held in memory during the window
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-all-errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      - name: keep-slow
        type: latency
        latency: { threshold_ms: 10000 }

      - name: keep-truncated
        type: string_attribute
        string_attribute:
          key: gen_ai.response.finish_reasons
          values: [length]

      - name: keep-expensive
        type: numeric_attribute
        numeric_attribute:
          key: gen_ai.usage.output_tokens
          min_value: 4000

      - name: keep-canary
        type: string_attribute
        string_attribute: { key: app.release_channel, values: [canary] }

      - name: keep-small-tenants     # low volume, so 100% is still cheap
        type: string_attribute
        string_attribute: { key: app.tenant_tier, values: [trial, pilot] }

      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

Three notes on making that work in practice. decision_wait must be longer than your slowest trace or long requests are evaluated incomplete and systematically dropped — for LLM traffic 15 seconds is a floor, not a default. The memory held is roughly expected_new_traces_per_sec × decision_wait × bytes_per_trace, so raising the window is a capacity decision. And every attribute a policy keys on must be set on the root span or be visible to the collector, which in practice means setting your attribution dimensions early rather than on the model span alone.

What is never sampled

• Metrics. Counters and histograms are pre-aggregated; their cost does not scale with request volume, only with label cardinality. Every rate, percentile and total you alert on should come from here, never from sampled traces.
• The request log row. One narrow row per request, content excluded, is small enough to keep in full for a year and is what makes cost attribution and per-tenant billing exact rather than estimated. Sampling this is a false economy.
• Errors, in any store. Whatever the rate, keep all of them. This is the cheapest thing on the list.
• Anything flagged by a user. A thumbs-down or a reported answer should pin its trace and its content beyond the normal TTL. These are your labelled examples and they arrive at perhaps one in a thousand.

What can be sampled aggressively is the expensive part: full message content, tool arguments, retrieved documents. Those dominate storage by an order of magnitude and are useful for a small fraction of investigations. A reasonable split is 100% of metadata, 100% of errored and flagged content, and a few percent of successful content for spot checks.

That split has a pleasant side effect on the privacy question. Content is the part with legal weight, and sampling it hard means the volume of personal data you hold is a few percent of what a naive “log everything” policy would accumulate — while the metrics, costs and error analysis, none of which need content, remain exact. Sampling and data minimisation turn out to be the same decision approached from two directions, which is a rare case of the cheap answer also being the careful one.

One last thing to decide explicitly rather than by default: the sampling policy is part of your incident response capability. If the policy drops everything except errors, an incident where requests succeed and answers are wrong leaves you with the 2% baseline sample and nothing else. Adding a switch that raises the baseline rate temporarily — turned on at the start of an investigation, off at the end — costs one flag and is worth having before you need it.