1 min lesson
Randomization unit: why request-level leaks
Give a practical answer to this: "You want to test a change to Cursor's Tab completion model. An engineer proposes randomizing at the individual completion-request level to maximize sample size. What is the core problem and what unit would you use instead?"
Step 1 of 2
Randomization unit: why request-level leaksthe most common trap in an editor experiment
Randomizing at the request or keystroke level feels efficient because you get more units. It quietly breaks the experiment. A single developer's session spans thousands of requests and their behavior in one request depends on what the model did in the last one. Split at the request level and the same person experiences both arms, so the arms contaminate each other and your treatment effect collapses toward zero.
- Unit
- User
- When it is right
- Almost always for product changes; behavior carries across a session
- The failure mode
- Fewer units than requests, so you need more calendar time
- Unit
- Workspace / repo
- When it is right
- Team-level features, shared context, seat-based effects
- The failure mode
- Heavy clustering inflates variance if you ignore it
- Unit
- Session
- When it is right
- Short-lived, stateless surfaces with no cross-session memory
- The failure mode
- Leaks if the change has any sticky or learning effect
- Unit
- Request
- When it is right
- Stateless system-level swaps that cannot affect later requests
- The failure mode
- Within-user contamination; rarely valid for product UX
| Unit | When it is right | The failure mode |
|---|---|---|
| User | Almost always for product changes; behavior carries across a session | Fewer units than requests, so you need more calendar time |
| Workspace / repo | Team-level features, shared context, seat-based effects | Heavy clustering inflates variance if you ignore it |
| Session | Short-lived, stateless surfaces with no cross-session memory | Leaks if the change has any sticky or learning effect |
| Request | Stateless system-level swaps that cannot affect later requests | Within-user contamination; rarely valid for product UX |
Default to user-level for product changes; justify anything coarser or finer explicitly.