Experimentation & Causal Inference

An A/B test is a chain of decisions made before you ever see a number. The interviewer is checking whether you make those decisions in the right order and can defend each one.

At Cursor the unit of analysis is the user-AI interaction, billions per period and the product is non-deterministic. That changes the texture of experiment design but not its skeleton. A rigorous test starts from a sharp hypothesis and walks down to a duration you can name out loud.

1. 1Hypothesis. A falsifiable statement with a direction: "capping agent tool-call retries at 3 will cut p95 turn latency without lowering task-completion rate." Vague hypotheses produce vague metrics.
2. 2Primary metric + guardrails. One metric you are powered to move (the primary metric), plus 2-3 guardrails you refuse to harm (completion rate, error rate, daily active edits). Add counter-metrics for the obvious gaming risk.
3. 3Randomization unit. Pick the level at which exposure is independent. For an editor this is almost always the user, occasionally the workspace, rarely the request.
4. 4Power, MDE, sample size. Decide the smallest effect worth detecting (the MDE), then solve for n at your variance and a fixed alpha/beta. The MDE is a product decision, not a statistics one.
5. 5Duration. n divided by daily eligible traffic, rounded up past at least one full weekly cycle so weekday/weekend mix and novelty wash out.

Randomization unit: why request-level leaksthe most common trap in an editor experiment

Randomizing at the request or keystroke level feels efficient because you get more units. It quietly breaks the experiment. A single developer's session spans thousands of requests and their behavior in one request depends on what the model did in the last one. Split at the request level and the same person experiences both arms, so the arms contaminate each other and your treatment effect collapses toward zero.

Peeking and sequential testingthe bug that fakes wins

A fixed-horizon test promises one significance level if you look once, at the end. Engineers in a hurry look every morning and ship the moment p drops below 0.05. That naive repeated peeking can inflate the false-positive rate past 30 percent. You either commit to a fixed sample and look once or you use a method built for continuous monitoring.

Sanity checks before you read resultstrust nothing until these pass

• Sample-ratio mismatch (SRM). If you assigned 50/50 but observe 50.8/49.2 at billions of units, a chi-square test flags it. SRM means the randomization or logging is broken and the whole readout is suspect.
• A/A test. Run two identical arms before or alongside the real test; a significant "effect" between them exposes a pipeline or variance-estimation bug.
• Instrumentation parity. Confirm both arms log the same events, the same way; a treatment that adds a new code path often adds new logging too, which silently shifts a metric.

On reliability data, the bottleneck is rarely the effect being small. It is the variance being enormous. Cutting variance is how you ship a real result in two weeks instead of two months.

Latency and reliability metrics are heavy-tailed: most turns are fast, a few are catastrophic and the slow tail dominates the mean and its variance. Required sample size scales with variance, so a method that shrinks variance directly shrinks the n and the calendar time you need. This is the single most impactful skill on a performance DS team.

CUPED and covariate adjustmentthe workhorse

CUPED (controlled-experiment using pre-experiment data) uses each user's pre-period metric as a covariate to remove the part of their outcome that was predictable before the experiment started. Because the pre-period value is unaffected by treatment, subtracting its influence cannot bias the estimate; it only strips out noise. A user who was already slow stays accounted for, so the residual variance you test on is smaller.

theta = cov(Y, X_pre) / var(X_pre)
Y_adj  = Y - theta * (X_pre - mean(X_pre))
# variance reduction ~ correlation(Y, X_pre)^2
# a pre-period that correlates 0.7 with the outcome
# removes ~49% of the variance -> ~half the required n

• The variance reduction equals the squared correlation between the pre-period covariate and the outcome, so pick a covariate that genuinely predicts the metric.
• Stratification and regression adjustment (CUPAC, ML-predicted controls) generalize the same idea to richer covariates.
• Adjustment never fixes a biased experiment; it makes an unbiased one more sensitive.

Taming heavy tailswhy the mean lies

Triggered analysis and the right estimatorcount the right users, with the right standard errors

• Trigger. Only include users who actually hit the changed code path. If 3 percent of users ever reach the modified agent retry path, diluting the effect across the other 97 percent destroys power. Define the trigger precisely and apply it identically to both arms.
• Ratio metrics. For metrics like success-per-attempt the user is the unit but the metric is a ratio of sums; use the delta method to get correct variance rather than treating it as a simple mean.
• Clustered errors. Session- or request-level rows from the same user are correlated; cluster standard errors at the randomization unit or you will dramatically understate uncertainty.

Most reliability questions are not A/B-able. A model swap, an infra migration or a region-wide incident hits everyone at once. There is no clean control arm to assign, so you reconstruct the counterfactual instead.

Quasi-experimental methods estimate "what would have happened without the change" from observational structure. Each one buys an identifying assumption and the interviewer's real test is whether you name that assumption and how you would falsify it. Match the method to the structure of the problem, not to your favorite tool.

Diff-in-diff in practicethe default reach for infra changes

You migrate one region's agent backend and leave a comparable region untouched. Diff-in-differences subtracts each region's pre-period level and compares the post-period change between them, which cancels out fixed differences between regions and any shock that hit both equally. The estimate is only causal if the two regions were trending in parallel before the migration.

metric ~ treated + post + (treated * post) + region_FE + day_FE
# the coefficient on (treated * post) is the causal estimate
# always plot the pre-period trends first:
# if they diverge before treatment, parallel-trends fails
# and the DiD number is not credible

Some changes can't be split per user at all. A shared inference pool, a retrieval index, a global rate limiter: there is one system and everyone uses it. The design has to move to the unit that can actually be isolated.

These are the designs that distinguish a senior experimentation DS from someone who only knows two-arm user tests. Each one exists to handle a structure where the simple split leaks or is infeasible.

Switchback experimentsrandomize time, not users

A switchback flips the whole system between treatment and control on a schedule (every 30 minutes, say) and treats each time bucket as a unit. It is the right call for system-level changes where a user-level split is impossible or would leak through shared infra. Because adjacent buckets are correlated and demand has daily rhythm, you balance the schedule across time-of-day and cluster standard errors at the bucket level.

Interleavingtwo retrieval variants on one screen

For ranking and retrieval, interleaving merges results from two variants into a single list and watches which side's items the user actually engages with. The same user judges both, which removes between-user variance and makes it dramatically more sensitive than a split test for comparing, say, two context-retrieval strategies for the agent. It tells you which variant wins, not how much a metric will move in production, so it is a fast screen rather than a launch decision.

Network and spillover effectsthe arms aren't independent

• On shared infra, the treatment arm's load can degrade the control arm (or vice versa), so the control is no longer a clean baseline. This is interference and it biases the estimate.
• The fix is to randomize at the isolating boundary: a separate inference pool, a region, a cluster, a time bucket, whatever cleanly separates one arm's resource contention from the other's.
• If you can't isolate, a switchback at least keeps both states from running simultaneously on the same pool.

Novelty, primacy and pre-launch evidenceshort tests mislead with developer tools

The deep-dive round isn't a method quiz. It's a pressure test of whether your causal claim survives someone actively trying to break it. Truth-seeking, the value Cursor screens for, looks like welcoming that pressure.

A strong answer has a shape. You state the counterfactual, name your identifying assumption, list the confounders and where they push, then triangulate across methods. When the evidence is thin, you say so and name the experiment that would settle it.

1. 1State the counterfactual. Say out loud what would have happened absent the change. "Latency dropped 12 percent" means nothing until you specify against which baseline.
2. 2Name the identifying assumption. Parallel trends, ignorability or the exclusion restriction, depending on the method, plus exactly how you'd test or falsify it.
3. 3Enumerate confounders and their direction. List what else changed and for each say whether it inflates or deflates your estimate. Direction of bias is more convincing than a vague hedge.
4. 4Triangulate. Show the effect holds across an A/B, a diff-in-diff and a back-of-envelope. Agreement across independent methods is far stronger than one tight p-value.
5. 5Bound your confidence. If you can't conclude causally, say it and propose the experiment to run next.

Correlation, causation, reverse causationname which one you're claiming