2 min lesson
Variance reduction & sensitivity
Answer this as if it were happening now: "CUPED uses a user's pre-experiment metric as a covariate. Why does this reduce variance without introducing bias and what determines how much variance it removes?" Say what supports your choice.
Step 1 of 3
On reliability data, the bottleneck is rarely the effect being small. It is the variance being enormous. Cutting variance is how you ship a real result in two weeks instead of two months.
Latency and reliability metrics are heavy-tailed: most turns are fast, a few are catastrophic and the slow tail dominates the mean and its variance. Required sample size scales with variance, so a method that shrinks variance directly shrinks the n and the calendar time you need. This is the single most impactful skill on a performance DS team.