Bootstrap Resampling, Done by Hand First

The bootstrap answers “how much would this number move if I collected the data again?” without collecting the data again and without assuming any distribution. You resample your own dataset with replacement, recompute the statistic each time, and read the spread off the results.

One dataset, and a question about many

Ten latency measurements, in milliseconds:

[120, 135, 140, 155, 160, 175, 190, 210, 260, 900]

mean   = 2445 / 10 = 244.5 ms
median = (160 + 175) / 2 = 167.5 ms

You want a confidence interval on that mean. The textbook route assumes the sampling distribution is roughly normal:

sample sd = 233.92
SE(mean)  = 233.92 / sqrt(10) = 73.97

95% CI = 244.5 +/- 1.96 * 73.97
       = 244.5 +/- 145.0
       = [99.5, 389.5]

The lower bound is 99.5 ms, which is below every measurement in the dataset. That is the normal approximation telling you it does not apply: one value of 900 has skewed everything, ten points is far too few for the central limit theorem to have taken hold, and latency distributions are not symmetric to begin with.

Two resamples, drawn by hand

A bootstrap resample draws ten values from these ten, with replacement, so some appear twice and some not at all. Here are two draws:

Resample 1: [140, 900, 155, 120, 900, 175, 260, 135, 160, 190]
  900 drawn twice, 210 not drawn
  sum  = 3135
  mean = 313.5

Resample 2: [120, 135, 140, 155, 160, 175, 190, 210, 260, 260]
  260 drawn twice, 900 not drawn
  sum  = 1805
  mean = 180.5

Two resamples of the same data give means of 313.5 and 180.5. That spread — produced entirely by which of your own observations happened to be drawn — is the bootstrap estimate of how much your statistic depends on the luck of which ten requests you happened to measure.

Do it ten thousand times, sort the ten thousand means, and take the 2.5th and 97.5th percentiles. That is a 95% bootstrap confidence interval, and it required no assumption about the shape of anything.

Notice what the two draws also reveal about this particular dataset. The mean is almost entirely determined by how many times the 900 is drawn, so the bootstrap distribution will be lumpy rather than smooth. That is not a defect of the method; it is the method reporting, correctly, that a mean over ten points with one large outlier is not a stable quantity.

Why sampling with replacement is the right move

Your sample is the best available estimate of the population. So treat it as the population, and draw new samples of the same size from it. Drawing without replacement would just return the same ten values in a different order and give zero variation.

A derivable consequence, which is where the bootstrap’s best-known number comes from:

Probability a given observation is missed by one draw:  1 - 1/n
Probability it is missed by all n draws:               (1 - 1/n)^n

n =  10:  (0.9)^10      = 0.3487
n = 100:  (0.99)^100    = 0.3660
n -> inf:  1/e          = 0.3679

So about 36.8% of observations are absent from any given
resample, and about 63.2% are present. That 63.2% is the
"in-bag" fraction, and it is also where random forests get
their out-of-bag error estimate.

The whole method, in twenty lines

import numpy as np

def bootstrap_ci(data, statistic=np.mean, n_boot=10_000,
                 alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    data = np.asarray(data, dtype=float)
    n = len(data)

    stats = np.empty(n_boot)
    for i in range(n_boot):
        resample = rng.choice(data, size=n, replace=True)
        stats[i] = statistic(resample)

    lo = np.percentile(stats, 100 * alpha / 2)
    hi = np.percentile(stats, 100 * (1 - alpha / 2))
    return statistic(data), lo, hi, stats

latencies = [120, 135, 140, 155, 160, 175, 190, 210, 260, 900]

for name, fn in [("mean", np.mean),
                 ("median", np.median),
                 ("p95", lambda x: np.percentile(x, 95))]:
    point, lo, hi, _ = bootstrap_ci(latencies, fn)
    print(f"{name:7s} {point:8.1f}   95% CI [{lo:.1f}, {hi:.1f}]")

1. Resample the same size as the original. Drawing fewer points would overstate the uncertainty; the sample size is part of what you are modelling.
2. Use at least 2,000 replicates for a 95% interval, and 10,000 if you care about the endpoints. The percentile estimate at the tails is itself noisy with too few.
3. Resample the unit you actually sampled. If you measured 50 requests across 5 users, resample users, not requests — otherwise you are assuming the 50 requests were independent when they were not, and the interval comes out far too narrow.
4. Fix the seed and report it. A bootstrap interval is itself a random quantity. Two runs give slightly different endpoints and somebody will notice.

The case where it is the only option

For a mean, there is a formula and the bootstrap is a convenience. For a median, a 95th percentile, a ratio of two medians, or a rank-correlation between two rankings, there is generally no usable closed form at all — and the bootstrap does not care.

Statistics with an easy formula:
  mean, proportion, difference of two means

Statistics without one:
  median
  p95 latency
  ratio of medians between two model routes
  "accuracy on the subset where retrieval succeeded"
  Spearman correlation between two leaderboards
  the win rate of A over B under a judge model

The bootstrap handles all of the second list identically:
resample, recompute, take percentiles.

This is why it is the right default for anything measured on production traffic. A p95 latency is exactly the kind of statistic people quote without an error bar because nobody knows how to compute one, and the answer is twenty lines and no theory. The same applies to an interval on a benchmark score, where the bootstrap and the closed-form binomial interval agree closely — which is a useful check that your harness is doing what you think.

Where the bootstrap fails

• It cannot fix a biased sample. The bootstrap estimates sampling variability, not systematic error. If your ten latency measurements all came from one region at one time of day, no amount of resampling will tell you about the other regions. The interval will be narrow and confidently wrong.
• It fails for extreme-value statistics. The maximum of a resample can never exceed the maximum of the original data, so a bootstrap interval on the maximum is bounded above by an observation and is systematically too low. The same problem afflicts very high percentiles on small samples — a p99 from 50 points is being estimated from at most one observation.
• It needs enough data to be worth resampling. With n = 5 there are limited distinct resamples and the interval is coarse. It is not wrong, but it is not informative either, and the honest report is the five numbers.
• It assumes independence between observations. Time-series data, repeated measurements on the same user, or benchmark items drawn from the same source document all violate it. The fix is to resample blocks or groups rather than individual points, and it must be a deliberate choice.
• The percentile interval is the simplest, not the best. For a skewed statistic the BCa (bias-corrected and accelerated) interval is more accurate, and every serious statistics library implements it. The percentile version is what to use when you are reading the code to check it, which is most of the time.