1 min lesson
A/B testing on millions of users
Use "A/B testing on millions of users" to explain each part and the role it plays.
Step 1 of 2
A/B testing on millions of users
Once a change clears offline, it earns a slice of live traffic. At Cursor's scale the statistics are easy; the discipline is in guardrails and honest readouts.
- Primary metric
- Pick one quality metric to win on (e.g. retention-weighted acceptance), decided before launch, not after.
- Guardrail metrics
- Latency p99, cost per completion, error rate - a win on quality that tanks latency is not a win.
- Ramp
- Start at 1%, watch guardrails, expand. A bad change should hurt thousands, not millions.
- Significance + duration
- Run long enough for novelty effects to fade and for the sample to be real, not until the number looks good.
The acceptance-vs-quality divergence is the failure that catches good teams. A model that suggests bigger, bolder completions can win acceptance and lose retention and if acceptance is your only metric you'll ship a regression that feels like progress. Always name the guardrail metric that would catch it.
Tie this straight back to the JD. When asked how you'd improve a completion model, answer in the loop the role actually runs: define a quality metric, gate on offline evals, ramp an A/B from 1% with latency and cost guardrails and read retention alongside acceptance. That's the literal job described as a method.