Skip to content

Log Sampler for Analysis

Draw a reproducible random sample from a large log file, with the sample size derived from the precision you need.

Draw this many
8

100.0% of 8 records, drawn uniformly at random without replacement.

Records in the file
8
Sample needed for ±5pp at 95%
8
Margin this sample actually buys
±0.00pp
Rarest event with a 95% chance of appearing
1 in 3
Rarest event with a coin-flip chance
1 in 12
Estimated tokens in the sample
≈ 380
The sample, in file order
What this assumes: the sample is uniform and without replacement, drawn with a seeded PRNG (mulberry32) so the same seed and file always give the same rows. The sample-size formula is the one for estimating a proportion at the worst case p = 0.5, with the finite-population correction n = n₀ / (1 + (n₀−1)/N) applied — so it is the right size for “what share of these requests did X”, and the wrong tool for “find me the rare failure”. Records are lines; a pretty-printed JSON log with one object across many lines needs flattening to JSONL first. The token figure is an estimate. Estimated, not tokenized. A real count needs the model's vocabulary; the authoritative number is usage.prompt_tokens on the response.

What the two numbers under the sample are for

Sampling a log is not one job, it is two, and they want opposite sample sizes. If the question is what proportion of these calls did something — hit the cache, exceeded a latency budget, returned a refusal — then a few hundred records answers it to within a few percentage points, however many billions the file holds. That is what the margin-of-error row is telling you, and it is why the required sample stops growing once the population is large: the correction term only matters when the sample is a big fraction of the whole.

If the question is does this rare failure still happen, the same sample is close to worthless, and the third row says how worthless. With 200 records drawn at random, an event occurring once in ten thousand requests has a 2% chance of showing up at all. Seeing zero of them proves nothing. That is not a defect in the sampling; it is what the arithmetic 1 − (1−p)ⁿ says, and the fix is a filter rather than a sample — take every record matching a predicate, and sample only the boring majority.

The seed is in the URL on purpose. Sampling is the step in an analysis most likely to be quietly re-run until the answer looks better, so a reviewer should be able to open your link, load the same file and get byte-identical rows. If they cannot reproduce your sample, they cannot check your conclusion.

Log Sampler for Analysis · Multigrid