1 min lesson
Triggered analysis and the right estimator
Use the lesson to respond to this: "A change touches a code path that only 4 percent of users ever reach, but you analyze the effect across all users and see nothing. What went wrong and what do you do?" Keep the answer plain.
Step 1 of 2
Triggered analysis and the right estimatorcount the right users, with the right standard errors
- Trigger. Only include users who actually hit the changed code path. If 3 percent of users ever reach the modified agent retry path, diluting the effect across the other 97 percent destroys power. Define the trigger precisely and apply it identically to both arms.
- Ratio metrics. For metrics like success-per-attempt the user is the unit but the metric is a ratio of sums; use the delta method to get correct variance rather than treating it as a simple mean.
- Clustered errors. Session- or request-level rows from the same user are correlated; cluster standard errors at the randomization unit or you will dramatically understate uncertainty.
Watch out
Powering for a rare failure is brutal. To detect a relative change in a 0.5 percent timeout rate, you may need millions of users. Before accepting that n, ask whether a smarter, denser proxy metric (time-to-first-token, a continuous reliability score) captures the same harm with far less variance. Picking a better metric often beats waiting for a bigger sample.