1 min lesson
Part 3: the curveballs they'll throw
Take this case and make the call. "You're running an A/B on a retrieval change and you check the dashboard daily. On day 3 the success-rate lift hits p = 0.04. The interviewer asks if you'd call it. What do you say?"
Step 1 of 2
Part 3: the curveballs they'll throw
Assignment isn't 50/50 when it should be, and a chi-square test flags it.
It signals a bucketing or logging bug; don't analyze the test, fix the pipeline first.
Repeatedly checking and stopping at the first significant look inflates false positives.
Fix the N in advance or use a sequential test, such as always-valid p-values, built for continuous monitoring.
Users react to change itself; early lift fades or early dip recovers.
Run past the novelty window and inspect the effect's time path, not just the pooled average.
The cause is the heavy tail from the metric-design section.
Apply CUPED with pre-period covariates to cut variance, or winsorize or cap the tail.
When you state a causal claim, attach how you'd break it in the same breath. "This assumes parallel trends; I'd falsify it by checking the pre-period, and if it fails I'd abandon diff-in-diff." Volunteering the falsification test is a strong truth-seeking signal. A confident estimate with no stated assumption reads as someone who hasn't run these in anger.