Metric Design & Product Sense

Reliability is a feeling until you give it a number. This seat exists to turn that feeling into a metric system an engineer could query tomorrow.

Cursor measures itself in user↔AI interactions at billions-per-period scale and the product is non-deterministic. The first thing the metric-design round checks is whether you reach for a single vanity number or a structured family of metrics that hold each other honest. A strong answer names the family before it names any one metric.

Picking a defensible north-starone number, heavily qualified

A good reliability north-star bundles three things into one definition: an outcome users care about, a quality bar and a time bound. "Agent success rate" alone is too loose. Share of agent runs that reach an accepted edit within the latency budget, measured over a rolling 7-day window is something a query can return.

Notice what that definition refuses to do. It does not reward a run that finishes fast but produces a rejected edit and it does not reward a correct edit that arrived after the user gave up. The bar and the time bound are both load-bearing.

Decompose into a driver treethe north-star as a product of queryable inputs

A north-star you can only observe is hard to act on. A north-star you can decompose tells engineers exactly where to look when it moves. Break it into the harness stages that multiply into it.

Engineers on a Performance & Reliability team talk in SLIs, SLOs and error budgets. If you want them to adopt your metric, define it in their language.

The error budget is the part candidates skip and the part that makes the metric powerful. It converts a target into a shipping policy. When the budget is healthy, teams ship fast. When it is spent, the policy says slow down and stabilize. The DS who owns the SLI quietly influences release cadence.

Choosing the percentilewhy p50 lies and the tail tells the truth

Reliability pain lives in the tail. A p50 that looks great can hide a p99 where one user in a hundred waits 30 seconds and abandons. Those abandoners churn loudest and complain hardest.

Composing SLOs across the harnessthe chain is only as reliable as its weakest stage

An agent run is a pipeline: retrieval, then model, then tool-call, then apply. End-to-end reliability is the product of the stage reliabilities, not the average. Four stages at 99% each compose to roughly 96% end-to-end and that gap is where users feel the product fail.

end_to_end = retrieval * model * tool_call * apply
            = 0.99   * 0.99  * 0.99      * 0.99
            ≈ 0.961   # ~3.9% of runs fail somewhere

This math is why a single end-to-end SLO is not enough. You need per-stage SLIs so the weakest link is visible and an error budget allocated across stages so no one stage silently eats the whole budget.

You cannot grade a single agent run the way you grade a unit test. The same prompt can produce a brilliant edit once and a wrong one the next. Measure the population, not the run.

Cursor's product is non-deterministic by design, so the metric has to be statistical from the start. Define success as a rate over many runs - share of runs that reach an accepted, correct edit - and watch its distribution and variance, not a single trace.

Proxy signals when there is no ground truththe user is your grader, implicitly

Most runs have no labeled "correct" answer. The user's behavior after the run is the closest thing to a label you get. Read it carefully.

No single proxy is trustworthy alone. Undo can mean "wrong" or "I changed my mind." The pattern across signals, segmented and tracked over time, is what carries the truth.

Model regression vs. harness regressionpage the right team

When the success metric drops, the most valuable thing you can do is attribute it. A model regression means the outputs got worse. A harness regression means the loop got slower or flakier while the model is fine. They page different teams and conflating them wastes the on-call.

Offline eval vs. online telemetrytwo truths that sometimes disagree

Scope the ask before you build the metricclarify, then implement

Half the bad metrics come from a vague request answered too fast. "Build me a reliability chart" hides a dozen decisions: which population, which window, how to handle multiple user IDs, which domains to include. The strongest practitioners force those decisions to the surface before they write a line of query.

Cursor bakes this in. Plan modeA mode that makes no edits: it researches the codebase and produces an editable plan you review before any code changes. ships with an ask-clarification tool that surfaces questions before it builds - it triggers on its own when a prompt is vague, or you can invoke it explicitly. One DS leans on it hard enough that they wrote a skill to make it non-optional.

An aggregate metric is where pain goes to hide. The metric-design round will hand you a number that looks healthy and watch whether you trust it or interrogate it.

The reflex they want is segment before you conclude. A global success rate of 94% can mask a brand-new client version sitting at 60% or one model that is quietly failing on large repos. The aggregate is an average of experiences and averages drown out the people who are suffering.

Dimensions to slice by defaultbefore you report any reliability number

The four trapsnamed, with the tell for each

Rank by blast radius, not by countthe JD says this explicitly

One of this role's named responsibilities is to identify the experiences causing disproportionate user frustration and quantify their blast radiusHow much breaks if a change goes wrong; the scope of potential damage.. That phrasing is deliberate. The loudest log line is not the worst problem.

Defining a metric is the easy half. The skill the JD actually screens for is operationalizing one: making it instrumented, trusted and wired into a decision the org will not override.

This role's requirement is a track record of operationalizing a north-star across a product pillar - driving adoption and decisions from it, not just authoring a definition. The interview wants the second half. A metric nobody checks is a comment in a doc.

The path from definition to decisionthe order matters

1. 1Define precisely. Outcome, quality bar, time window - concrete enough to query tomorrow.
2. 2Build the data contract. Event schema, ownership, backfill and data-quality checks so the number stays trustworthy as the product changes.
3. 3Establish one source of truth. Put the metric in a semantic/metric layer so every dashboard derives from the same definition.
4. 4Wire it into a ritual. A weekly reliability review, a release-readiness gate, an incident retro template.
5. 5Close the loop. Show one case where the metric changed a decision or triggered a rollback, then keep that story.

Self-serve so you are not the bottlenecka named responsibility

Cross-functional empathy here means making engineers self-sufficient instead of gatekeeping data. The metric layer, the dashboards and the query interfaces exist so an engineer can answer their own reliability question at 2am without filing a ticket to you. The win is when people stop asking you and start asking the dashboard.

Spec the metric as a code-linked, reviewable docthe workflow, not just the definition

The old way to spec a metric change was a design doc in Notion or Drive, disconnected from the code it described. Cursor's plan mode loads the plan beside the code, so a step can point at the exact file and snippet it will touch. You edit the plan directly - delete a chart step and the agent drops it, add a step to ask for a test or a different cut. Check the plan into git and it becomes a reviewable paper trail your team can argue over before any query runs.

The same surface handles the messy front end of metric work. Cursor takes image inputs, so a team can photograph their hand-drawn architecture sketches and have it convert them to Mermaid - one customer published the result straight to FigJam. It is strong at pulling a sprawling Excel sheet into a notebook too: point it at a funding spreadsheet and ask for a Jupyter notebook querying that data. Hand it a creative, open-ended task and it reaches for a built-in brainstorming skill on its own.