Skip to lesson
Exit
Experimentation & Causal Inference1 / 3

2 min lesson

Interleaving

Use the evidence to decide "When would you choose interleaving over a standard A/B test and what is its main limitation?" Explain the choice plainly.

Step 1 of 3

Interleavingtwo retrieval variants on one screen

For ranking and retrieval, interleaving merges results from two variants into a single list and watches which side's items the user actually engages with. The same user judges both, which removes between-user variance and makes it dramatically more sensitive than a split test for comparing, say, two context-retrieval strategies for the agent. It tells you which variant wins, not how much a metric will move in production, so it is a fast screen rather than a launch decision.

Learn more

Full explanation

Network and spillover effects

Network and spillover effectsthe arms aren't independent

  • On shared infra, the treatment arm's load can degrade the control arm (or vice versa), so the control is no longer a clean baseline. This is interference and it biases the estimate.
  • The fix is to randomize at the isolating boundary: a separate inference pool, a region, a cluster, a time bucket, whatever cleanly separates one arm's resource contention from the other's.
  • If you can't isolate, a switchback at least keeps both states from running simultaneously on the same pool.