1 min lesson
Sanity checks before you read results
Recall the main items in "Sanity checks before you read results", then connect each one to the work.
Step 1 of 2
Sanity checks before you read resultstrust nothing until these pass
- Sample-ratio mismatch (SRM). If you assigned 50/50 but observe 50.8/49.2 at billions of units, a chi-square test flags it. SRM means the randomization or logging is broken and the whole readout is suspect.
- A/A test. Run two identical arms before or alongside the real test; a significant "effect" between them exposes a pipeline or variance-estimation bug.
- Instrumentation parity. Confirm both arms log the same events, the same way; a treatment that adds a new code path often adds new logging too, which silently shifts a metric.
Interview move
When handed a result, don't react to the number first. Ask whether SRM passed and whether the pipeline was A/A-clean before you read the effect. Leading with diagnostics signals you have run real experiments and have been burned by a broken one.
Watch out
"p < 0.05, ship it" is not an answer. Report the effect size with a confidence interval and ask whether the lower bound clears the practical-significance bar you set. A statistically significant 0.1 percent latency win that costs a week of eng time fails the decision rule you should have pre-registered.