1 min lesson
Robust statistics, not parametric defaults
Describe what "Reach for estimators that survive ugly distributions" changes in practice.
Step 1 of 3
Robust statistics, not parametric defaults
Reach for estimators that survive ugly distributions. Percentiles describe the experience directly: p50 is the typical request, p95/p99 are the tail you're paid to protect. When you need a center of mass, a trimmed mean or median beats the raw mean. For spread, MAD (median absolute deviation) shrugs off outliers that would blow up a standard deviation.
Learn more
Advanced table
Heavy-tailed data punishes parametric defaults
- Question
- Typical latency?
- Naive tool
- Mean
- Robust tool
- Median (p50)
- Question
- Tail experience?
- Naive tool
- Max
- Robust tool
- p95 / p99 (and watch p99.9)
- Question
- Spread?
- Naive tool
- Std dev
- Robust tool
- MAD or IQR
- Question
- Confidence interval on a skewed metric?
- Naive tool
- t-interval
- Robust tool
- Bootstrap
- Question
- Center when outliers present?
- Naive tool
- Mean
- Robust tool
- Trimmed mean
| Question | Naive tool | Robust tool |
|---|---|---|
| Typical latency? | Mean | Median (p50) |
| Tail experience? | Max | p95 / p99 (and watch p99.9) |
| Spread? | Std dev | MAD or IQR |
| Confidence interval on a skewed metric? | t-interval | Bootstrap |
| Center when outliers present? | Mean | Trimmed mean |
Heavy-tailed data punishes parametric defaults; robust estimators are the honest answer.
When a distribution is too gnarly for a closed-form interval, bootstrap it. Resample the data with replacement thousands of times, recompute your statistic each time and read the confidence interval off the resampled distribution. It makes no normality assumption, which is exactly why it fits latency and success-rate work.
import numpy as np
def bootstrap_p95(latencies, n_boot=10_000, alpha=0.05):
x = np.asarray(latencies, dtype=float)
boot = np.empty(n_boot)
for i in range(n_boot):
sample = np.random.choice(x, size=x.size, replace=True)
boot[i] = np.percentile(sample, 95)
lo, hi = np.percentile(boot, [100 * alpha / 2, 100 * (1 - alpha / 2)])
return float(np.percentile(x, 95)), (float(lo), float(hi))
# p95 with a 95% CI you can defend on skewed data