1 min lesson
Part 2 - “we can't randomize this model swap”
Say what this means in practice: "we can't randomize this model swap."
Step 1 of 3
Part 2 - “we can't randomize this model swap”
A model rollout often can't be A/B'd: it ships to everyone on a date or routing constraints prevent clean assignment. Now you need a quasi-experiment and the move is matching the method to the structure of the data you actually have.
Learn more
Advanced table
Reference table
- Method
- Diff-in-diff
- Use when
- A comparison group never got the swap (e.g., one region or client held back).
- Key assumption to falsify
- Parallel trends pre-swap - plot the two groups before the change and show they moved together.
- Method
- Synthetic control
- Use when
- No single clean control, but many candidate segments to weight into one.
- Key assumption to falsify
- The synthetic tracks the treated unit well in the pre-period; check fit on held-out pre-weeks.
- Method
- Regression discontinuity
- Use when
- Assignment hinges on a threshold (rollout to client_version ≥ X).
- Key assumption to falsify
- No other change happens at the same cutoff; units can't manipulate which side they're on.
- Method
- Interrupted time series
- Use when
- Everyone switched at once, with a clean before/after on a stable series.
- Key assumption to falsify
- No co-occurring deploy or seasonality at the breakpoint; model the counterfactual trend.
| Method | Use when | Key assumption to falsify |
|---|---|---|
| Diff-in-diff | A comparison group never got the swap (e.g., one region or client held back). | Parallel trends pre-swap - plot the two groups before the change and show they moved together. |
| Synthetic control | No single clean control, but many candidate segments to weight into one. | The synthetic tracks the treated unit well in the pre-period; check fit on held-out pre-weeks. |
| Regression discontinuity | Assignment hinges on a threshold (rollout to client_version ≥ X). | No other change happens at the same cutoff; units can't manipulate which side they're on. |
| Interrupted time series | Everyone switched at once, with a clean before/after on a stable series. | No co-occurring deploy or seasonality at the breakpoint; model the counterfactual trend. |
Pick by what natural control the rollout left you, then name the assumption you'd test before trusting the estimate.
“We held the swap back from one region for a week, so I'd reach for diff-in-diff. The whole thing rests on parallel pre-trends, so before I quote an effect I'd plot both regions for the eight weeks before the swap. If they weren't already moving together, diff-in-diff is the wrong tool and I'd switch to a synthetic control weighted on the pre-period.”