1 min lesson
When you can't randomize
Describe the practical point in "Most reliability questions are not A/B-able", then say what it changes.
Step 1 of 3
Most reliability questions are not A/B-able. A model swap, an infra migration or a region-wide incident hits everyone at once. There is no clean control arm to assign, so you reconstruct the counterfactual instead.
Quasi-experimental methods estimate "what would have happened without the change" from observational structure. Each one buys an identifying assumption and the interviewer's real test is whether you name that assumption and how you would falsify it. Match the method to the structure of the problem, not to your favorite tool.
Learn more
Advanced table
The method is only as good as the assumption you can defend and test
- Method
- Difference-in-differences
- Use when
- A clean comparison group exists (another region, client version) and the change hits one of them
- Identifying assumption
- Parallel trends: groups would have moved together absent treatment
- How to stress-test it
- Plot pre-period trends; run a placebo on a pre-treatment date
- Method
- Synthetic control
- Use when
- No single clean control; build a weighted blend of donors (good for staged rollouts)
- Identifying assumption
- The weighted donors reproduce the treated unit's pre-trend
- How to stress-test it
- Check pre-period fit; placebo-test on untreated donor units
- Method
- Regression discontinuity
- Use when
- A sharp threshold assigns the change (timeout cutoff, rollout-percent boundary)
- Identifying assumption
- Units just above and below the cutoff are comparable
- How to stress-test it
- Look for bunching/manipulation at the cutoff; vary the bandwidth
- Method
- Propensity matching
- Use when
- Observational groups differ on measured confounders
- Identifying assumption
- No unmeasured confounders (ignorability) given the covariates
- How to stress-test it
- Check covariate balance after matching; bound unmeasured bias
- Method
- Instrumental variables
- Use when
- An instrument shifts exposure but not the outcome directly
- Identifying assumption
- Relevance + exclusion: instrument affects outcome only through exposure
- How to stress-test it
- Test instrument strength (F-stat); argue exclusion qualitatively
| Method | Use when | Identifying assumption | How to stress-test it |
|---|---|---|---|
| Difference-in-differences | A clean comparison group exists (another region, client version) and the change hits one of them | Parallel trends: groups would have moved together absent treatment | Plot pre-period trends; run a placebo on a pre-treatment date |
| Synthetic control | No single clean control; build a weighted blend of donors (good for staged rollouts) | The weighted donors reproduce the treated unit's pre-trend | Check pre-period fit; placebo-test on untreated donor units |
| Regression discontinuity | A sharp threshold assigns the change (timeout cutoff, rollout-percent boundary) | Units just above and below the cutoff are comparable | Look for bunching/manipulation at the cutoff; vary the bandwidth |
| Propensity matching | Observational groups differ on measured confounders | No unmeasured confounders (ignorability) given the covariates | Check covariate balance after matching; bound unmeasured bias |
| Instrumental variables | An instrument shifts exposure but not the outcome directly | Relevance + exclusion: instrument affects outcome only through exposure | Test instrument strength (F-stat); argue exclusion qualitatively |
The method is only as good as the assumption you can defend and test.
Learn more
Optional practice
Practice: When you can't randomize
QCursor migrates the agent backend in one region and you want to know if it changed p95 latency. You plan a difference-in-differences against an untouched region. What single check most determines whether your estimate is causal?