1 min lesson
Part 3 - the curveballs they'll throw
Take this case and make the call: "You're running an A/B on a retrieval change and you check the dashboard daily. On day 3 the success-rate lift hits p = 0.04. The interviewer asks if you'd call it. What do you say?"
Step 1 of 2
Part 3 - the curveballs they'll throw
Assignment isn't 50/50 when it should be - a chi-square flags it.
It signals a bucketing or logging bug; don't analyze the test, fix the pipeline first.
Repeatedly checking and stopping at the first significant look inflates false positives.
Fix the N in advance or use a sequential test (e.g. always-valid p-values) built for continuous monitoring.
Users react to change itself; early lift fades or early dip recovers.
Run past the novelty window and inspect the effect's time path, not just the pooled average.
Latency's tail makes the mean estimate noisy and underpowers the test.
Use percentile metrics, winsorize or cap and apply CUPED to cut variance with pre-period covariates.
When you state a causal claim, attach how you'd break it in the same breath. “This assumes parallel trends; I'd falsify it by checking the pre-period and if it fails I'd abandon diff-in-diff.” Volunteering the falsification test is the truth-seeking signal Cursor screens for. A confident estimate with no stated assumption reads as someone who hasn't run these in anger.