1 min lesson
Metric-design case
Walk through "Metric-design case" in order, then name the proof that tells you it worked.
Step 1 of 2
“Define how we measure agent reliability for Cursor.” That's the prompt. It's deliberately open and the round grades whether you can turn an abstract word - reliability - into something a team could query tomorrow and gate a release on.
Run a fixed structure so the ambiguity doesn't sink you. State what you're measuring and for whom, propose the metric tree, then describe how it lives in the org. Reserve the last third for operationalization, because that's where this round separates a definition from a slide.
- 1Scope it (60s). “Reliability” for the agent harness means: does an agent run complete, correctly and fast enough that the user keeps going? Name what you're not measuring (model quality of the underlying LLM is a separate pillar).
- 2Pick the north-star. One metric that captures the experience and moves with real wins. Then the guardrails that must not degrade while you chase it and the counter-metrics that catch you gaming it.
- 3Make each metric queryable-tomorrow. Give the formula, the unit (logical attempt), the window and the exact event fields.
- 4Operationalize it. Dashboard, alert, release gate and the ritual that forces a decision.
Learn more
Advanced table
The metric tree, made concrete
The metric tree, made concrete
- Role
- North-star
- Metric
- Agent Success Rate
- Definition (queryable-tomorrow)
- Logical attempts that reach event_type='success' / total logical attempts, per request_id, 7-day trailing, segmented by model + client_version.
- Event fields
- request_id, event_type, status, model, client_version, ts
- Role
- Guardrail
- Metric
- p95 completion latency
- Definition (queryable-tomorrow)
- PERCENTILE_CONT(0.95) of latency_ms on successful attempts; must not regress > X% week-over-week.
- Event fields
- latency_ms, event_type, ts
- Role
- Guardrail
- Metric
- Tool-call reliability
- Definition (queryable-tomorrow)
- tool_call events returning ok / total tool_call events; agent loops die here silently when this slips.
- Event fields
- event_type, status, request_id
- Role
- Counter-metric
- Metric
- Avg tries per attempt
- Definition (queryable-tomorrow)
- AVG(tries) per request_id; rises when success is bought with aggressive retrying - catches a gamed success rate.
- Event fields
- request_id, event_type
- Role
- Counter-metric
- Metric
- Trivial-success share
- Definition (queryable-tomorrow)
- Share of successes on no-op / one-line edits; guards against inflating success by counting cheap interactions.
- Event fields
- edit size, event_type
| Role | Metric | Definition (queryable-tomorrow) | Event fields |
|---|---|---|---|
| North-star | Agent Success Rate | Logical attempts that reach event_type='success' / total logical attempts, per request_id, 7-day trailing, segmented by model + client_version. | request_id, event_type, status, model, client_version, ts |
| Guardrail | p95 completion latency | PERCENTILE_CONT(0.95) of latency_ms on successful attempts; must not regress > X% week-over-week. | latency_ms, event_type, ts |
| Guardrail | Tool-call reliability | tool_call events returning ok / total tool_call events; agent loops die here silently when this slips. | event_type, status, request_id |
| Counter-metric | Avg tries per attempt | AVG(tries) per request_id; rises when success is bought with aggressive retrying - catches a gamed success rate. | request_id, event_type |
| Counter-metric | Trivial-success share | Share of successes on no-op / one-line edits; guards against inflating success by counting cheap interactions. | edit size, event_type |
Every row names its fields so the panel can see you could write the query in the room.
Confront the three things that make this hard
The same prompt can succeed or fail across runs; a single trace proves nothing.
Measure rates over many attempts and treat a regression as a shift in the rate distribution, not one bad run.
Latency is right-skewed; the mean hides the slow experiences that drive frustration.
Define guardrails on p95/p99 and watch the tail separately from the median.
A flat success rate can hide a 20-point drop in one region or client_version (Simpson's paradox).
Every headline metric ships pre-segmented by model, client_version and region.