You have one sample and no formula for the uncertainty of its median. The bootstrap draws new samples from the sample itself and watches the median move. This lesson runs the bootstrap for a median and a trimmed mean, works out a small bootstrap exactly by counting, and shows where the method breaks.
This is the first lesson of Resampling, Power and Honest p-values: A Simulation Workshop, free for everyone to read.
Here are the commute times of nine workers at a small office, in minutes. The data are made up.
, , , , , , , and minutes.
The median is minutes. One worker lives far away, so the mean is pulled up to about minutes. We would like to report the median with a margin of error. In the free course on sampling variability, the standard error of a mean came from a formula. For a median there is no simple formula that works for a sample this small.
If we could, we would draw many new samples of nine workers from the same population and watch the median move. We cannot. We have one sample.
The bootstrap makes the sample stand in for the population. It works like this.
Why with replacement? Without it, every resample would be the same nine numbers in a new order, with the same median. With replacement, each draw is like picking one more worker from a population that looks like our sample.
The standard deviation of the resample medians is the bootstrap standard error. The values that cut off the lowest and the highest of the resample medians give a percentile interval, a interval for the median. The statistician Bradley Efron proposed the method in 1979. You may have met the bootstrap of a mean in the Python course on Simulation and Randomness.
The program draws resamples. For each one it works out the median, the mean and a trimmed mean, which drops the two smallest and the two largest values and averages the rest. It then prints the percentile interval for the median and the three bootstrap standard errors.
The mean moves most because the long commute sometimes appears two or three times in a resample and sometimes not at all. The median and the trimmed mean pay little attention to one extreme value.
For a median, we do not even need a simulation. Sort the sample, so that the values are . The median of a resample is at most exactly when at least of its draws are at most . Each draw is at most with chance . So the number of such draws is binomial, and
For this gives about , and for about . So the resample median equals the sample median, minutes, with chance about . The picture shows all nine chances.
Two things stand out. First, the chances depend only on the order of the values, not on their size. The long commute of minutes could be minutes and nothing would change. Second, the median of a resample can only be one of the nine sample values. With nine values, the bootstrap distribution is lumpy, and the percentile interval jumps from one sample value to the next.
The exact chance that the resample median is at most minutes is about , already above . The chance that it is at most minutes is about , still below . So the exact percentile interval runs from to minutes, just as the simulation found.
For a mean, the bootstrap has an exact answer too. With very many resamples, the bootstrap standard error of the mean is
where is the standard deviation found with the population formula, dividing by . The reason is that each draw of a resample is one value picked at random from the sample, and the sample, treated as a population, has standard deviation .
For the commute times, minutes, so minutes. The usual formula uses the sample standard deviation , dividing by , and gives minutes. The two differ by the factor , which is close to for a large sample but not for nine values.
The bootstrap is not magic. It can only show the variation that the sample itself contains.
A tiny sample. With five values, the percentile interval for a median can run from the smallest value to the largest. That tells you nothing new.
The largest value. Suppose you want to estimate the longest commute in the whole office. A resample can never contain a value larger than the sample's largest, minutes. Worse, the resample keeps that largest value with chance . So most resamples give exactly minutes, and the bootstrap claims a certainty it does not have.
What a resample leaves out. A given value is missing from a resample with chance . For large this settles near , so a typical resample holds only about of the different values.
A biased sample. If the nine workers were the nine who answered an email, the bootstrap resamples that biased sample. It gives a fine standard error around the wrong value. Resampling measures chance scatter. It never removes bias.
Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.
Resampling, Power and Honest p-values: A Simulation Workshop is a Standard course. It adds 2 more lessons, practice and tests on top of the free chapter.