Reliability Analytics & Self-Serve Tooling

The reliability mandate at Cursor lives or dies on one question: do you find a regression from a screenshot a user posts or from your own alert that fired twenty minutes after the deploy? This section is about building the second world.

A detection framework has two halves that get confused in interviews. Pre-deploy catches the regression before it reaches everyone, on a slice of traffic you control. Post-deploy catches what slipped through, on the live time series of latency, error rate and agent success. You need both, because a canary that sees 1% of traffic for ten minutes cannot surface a regression that only bites the long tail of a particular model on a particular client.

The two layers and what each one is forpre-deploy vs post-deploy

Start a canary by proving your measurement is trustworthy. An A/A test routes two arms to identical code and confirms the metric shows no difference. If your A/A flags a phantom regression, your detector is miscalibrated and every real alert is suspect.

1. 1A/A baseline. Split identical traffic two ways; confirm the latency and success deltas sit inside noise. This calibrates the false-positive rate you'll live with.
2. 2Canary the change. Route a small, representative slice to the new build. Representative matters more than large: cover the models, regions and clients that dominate volume.
3. 3Compare on guardrails. Watch p95/p99 latency, error rate, timeout rate and agent success on the canary versus control, not just the average.
4. 4Promote or roll back. Gate the full rollout on the canary clearing its thresholds. A failed canary is a non-event for users, which is the whole point.
5. 5Watch the live series. Once promoted, change-point detection takes over on the full population to catch what 1% couldn't.

Change-point detection over static thresholds

A fixed threshold like "alert if p95 > 4s" breaks both ways: it's deaf to a regression that moves p95 from 800ms to 1.6s and it screams every time legitimate traffic shifts. Change-point detection asks a better question - did the distribution's behavior shift at some point in time, relative to its own recent history?

Sensitivity vs alert fatigue

Every detector has a knob between catching real regressions and crying wolf. Tie that knob to economics, not vibes. The cost of a missed regression is its blast radiusHow much breaks if a change goes wrong; the scope of potential damage. in frustrated users; the cost of a false positive is an engineer's interrupted afternoon and slow erosion of trust in the alert. Calibrate against your error budget: page when burn rate threatens the SLO, file a ticket when it merely nicks it.

Segment-aware alerting

The global line is a liar. A regression that doubles latency for one model in one region can be completely invisible in the aggregate, because that slice is a small share of total volume. At Cursor's scale the unit of analysis is the user-AI interaction across billions per period and the regressions that matter most often hide in a single dimension.

• Cut every reliability metric by model, region, client version and surface (Tab, Agent, chat) before you trust the headline number.
• A flat global p95 with a spiking per-model p95 is the classic Simpson's-paradox trap - the mix moved, not the experience.
• Run detection per segment, not only on the rollup or you'll ship the regression that only hurts the new client build.

Make it actionable, then auto-triage

A detected regression that says "latency is up" wastes the on-call's first hour. Stamp every interaction with a release/commit/model-version dimension at log time so a change-point can name its likely cause: this regression starts at deploy abc123, on model X. That single join turns an observation into an assignment.

The hardest, most Cursor-specific triage is separating a model regression from a harness regression. The model got worse versus your agent loop, tool-calls or context assembly got worse. Route them to different owners and route fast.

Latency logs are not a tidy bell curve you can summarize with a mean and a standard deviation. They are heavy-tailed, multimodal, censored and polluted - and the candidate who averages them away will get caught in the technical screen.

A request that hits a warm cache returns in 40ms; the same request on a cold start with a slow tool-call takes 8 seconds. That's two populations in one column. The mean lands in the empty valley between them, describing no real request. Model the shape before you summarize it.

Why the mean lies hereshape before summary

Robust statistics, not parametric defaults

Reach for estimators that survive ugly distributions. Percentiles describe the experience directly: p50 is the typical request, p95/p99 are the tail you're paid to protect. When you need a center of mass, a trimmed mean or median beats the raw mean. For spread, MAD (median absolute deviation) shrugs off outliers that would blow up a standard deviation.

When a distribution is too gnarly for a closed-form interval, bootstrap it. Resample the data with replacement thousands of times, recompute your statistic each time and read the confidence interval off the resampled distribution. It makes no normality assumption, which is exactly why it fits latency and success-rate work.

import numpy as np

def bootstrap_p95(latencies, n_boot=10_000, alpha=0.05):
    x = np.asarray(latencies, dtype=float)
    boot = np.empty(n_boot)
    for i in range(n_boot):
        sample = np.random.choice(x, size=x.size, replace=True)
        boot[i] = np.percentile(sample, 95)
    lo, hi = np.percentile(boot, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return float(np.percentile(x, 95)), (float(lo), float(hi))

# p95 with a 95% CI you can defend on skewed data

De-noise before you count

Raw logs over-count failures and under-count truth. Three retries of one user action look like three failures and one success unless you collapse them. Client clock skew makes some latencies negative. Bot and automation traffic has a different distribution than humans. An instrumentation bug can manufacture a regression that never touched a user.

1. 1Sessionize. Group events into logical attempts so one user action is one row, not one row per retry.
2. 2Dedup retries and timeouts. Collapse a retry chain into a single attempt with a final outcome before you compute failure rate.
3. 3Filter non-human traffic. Segment out bots and automation or at least tag them so they never silently inflate a metric.
4. 4Sanity-check the clock. Drop or correct negative and absurd durations from client skew before they poison percentiles.
5. 5Audit the instrument. When a metric jumps, rule out an instrumentation or schema change before declaring a real regression.

Censoring and survivorship

An aborted or still-running agent turn is right-censored: you know it took at least this long, not how long it finally took. Treating it as a completed fast request or dropping it, both bias you toward looking healthier than you are. Survivorship bias is the same trap from the other side: if only successful runs reach the table you analyze, your success rate is a tautology.

The technical screen for this role is data-shaped, not algorithmic: SQL on event/interaction data plus applied stats. They're checking whether you can actually pull p95 latency by model from a billion-row table without melting the warehouse - and whether someone could rerun your number tomorrow.

Three skills carry most screens. Window functions for sessionization and funnels, percentile aggregations for latency and cost-aware querying so your scan touches partitions instead of the whole table. Show all three on one realistic problem and you've answered the round.

Window functions and percentiles in angerthe moves they probe

WITH events AS (
  SELECT user_id, model, run_id, event_ts, latency_ms, outcome
  FROM agent_interactions
  WHERE event_ts >= CURRENT_DATE - INTERVAL '7' DAY   -- partition prune
    AND surface = 'agent'
),
latency_by_model AS (
  SELECT
    model,
    APPROX_PERCENTILE(latency_ms, 0.50) AS p50,
    APPROX_PERCENTILE(latency_ms, 0.95) AS p95,
    APPROX_PERCENTILE(latency_ms, 0.99) AS p99,
    AVG(CASE WHEN outcome = 'success' THEN 1.0 ELSE 0.0 END) AS success_rate,
    COUNT(*) AS n
  FROM events
  GROUP BY model
)
SELECT * FROM latency_by_model
WHERE n >= 1000          -- ignore segments too small to trust
ORDER BY p95 DESC;

Two craft signals are hiding in that query. The date filter on a partitioned column prunes the scan instead of reading every row. And APPROX_PERCENTILE (or percentile_cont where exactness is needed) computes the tail at scale where an exact sort would be prohibitive - naming the exact-vs-approximate tradeoff out loud is a senior tell.

Cost-aware querying on billions of rows

Python for log analysis and detection

SQL gets you aggregates; Python gets you the modeling SQL can't express. Pull a manageable slice with pandas or polars, run hypothesis tests with scipy/statsmodels and prototype lightweight detection - a change-point or anomaly check - before anyone builds the production version.

from statsmodels.stats.proportion import proportions_ztest

# successes / totals for control (pre) and treatment (post)
successes = [pre_success, post_success]
totals    = [pre_total,   post_total]

stat, pval = proportions_ztest(count=successes, nobs=totals)
delta = post_success / post_total - pre_success / pre_total
print(f"success-rate delta={delta:+.3%}  p={pval:.4f}")
# A real regression should clear significance AND be large enough to matter

Driving Cursor through the analysis itselfthe analyst's own loop

This is also the work you'll do inside Cursor day to day, and it pays to set the workspace up like the team does. Cursor ships role-specific cookbooks - workflow recipes for different jobs - and the data science one is the place to start. It walks through Jupyter notebook development, wiring up database frameworks like Supabase and Postgres and the in-editor extensions for Postgres, BigQuery, SQLite and Snowflake so your warehouse lives next to your analysis.

For notebooks, install the Jupyter extensions from the same marketplace VS Code uses - check the publisher (the ones by MS Tools AI) and the download count before you trust one. One honest caveat from the field: Cursor can't always run Jupyter cells for you automatically, so expect to execute some cells by hand. The flip side is that Python frustrations are where the agent shines. Because Python is so heavily represented in training data, models are strong at the pip-version-venv mess - one recurring fix is pandas import could not be resolved because the package sat in system Python but not in the venv the notebook was using, which Cursor will diagnose, install into the venv and self-verify.

Model economics: plan with the smart model, build with the cheap one

The single rule of thumb worth stealing from the team: don't reach for the most expensive reasoning model by reflex. Start with the cheaper, standard model - the in-house ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan. models are fast and genuinely underestimated - and only escalate to a frontier reasoning model (GPT-5.x, Opus 4.x) when the cheap one stalls. The exception is when a task obviously needs reasoning: building a plan, or a thorny bug where the model has to reason about where the problem even lives. Skip straight to the strong model there.

When you batch out work - say, generating a wall of charts from a cleaned export - reach for sub-agents. They're "baby Cursors": smaller agents with a clear persona and instructions, running inside a bigger agent, spun up from a create sub agent skill (roughly a 150-line Markdown persona). The Cursor-specific lever is that you set a model per sub-agentA child agent a main agent spawns to work in parallel with its own context window, handing results back so the parent's context stays clean., so a chart-builder doesn't need a frontier model racking up a bill - one of the in-house ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan. models or a mid-tier model is plenty. You can launch several in parallel ("launch three chart-builder sub-agents" spins up three side-by-side workers, each visible in the sub-agent UI). One hard constraint: sub-agents need a reasoning model to drive them - the fastest execution-only Composer variant can't run sub-agents, so point them at a reasoning model.

Reproducibility and data quality

An analysis an engineer can't rerun is a rumor. Parameterize your queries, keep the analysis under version control and make the result regenerate from inputs on demand. Then bake data-quality checks into the pipeline itself so a broken metric fails loudly instead of lying quietly.

• Parameterize date ranges, segments and thresholds - no magic numbers buried in SQL.
• Version-control the analysis so the number has provenance and a diff.
• Assert null, duplicate and skew checks in-pipeline; a metric that silently drops 30% of rows is worse than no metric.
• Pin the query and inputs so "rerun it next month" gives the same shape of answer, not a surprise.

The JD is explicit: you build data tooling that lets engineers answer their own questions, instead of routing every reliability query through you. The win condition is fewer ad-hoc asks, not more dashboards.

This is the difference between a data scientist who gatekeeps and one who scales the org. Cursor wants the second kind. The core artifact is a metric layer that makes "agent success rate" mean one thing everywhere, plus interfaces an engineer can drive without a DS in the loop.

The semantic / metric layerdefine once, agree everywhere

Without a metric layer, every team writes its own SQL for "success rate," and three dashboards disagree by lunch. A semantic layer defines each metric once - the exact filter, the denominator, the sessionization - so every dashboard, alert and query resolves to the same definition. It kills the "whose number is right" debate that quietly drains trust.

metric: agent_success_rate
  description: Share of admitted agent runs that reach a success outcome
  source: agent_interactions
  entity: run_id                 # sessionized: one row per logical attempt
  filter: surface = 'agent' AND is_bot = false
  numerator:   count(run_id) where outcome = 'success'
  denominator: count(run_id)     # admitted attempts, includes censored = failure
  dimensions: [model, region, client_version, surface]
  owner: reliability-ds

Dashboards and query interfaces engineers actually use

For the one-off end of the spectrum - a finding you want to see right now, not maintain - Canvas is the fast path. Cursor writes React to render a dashboard directly inside the IDE, so an anonymized CSV usage export becomes a live, clickable dashboard you iterate on by talking to it ("match the Cursor dashboard", "add a tooltip"). It takes an image, a CSV or the codebase as input. The decision rule keeps it from becoming shadow tooling: Canvas is for one-off visualization and sharing. If you want an always-on dashboard, have Cursor build a real website - a small Next.js app - and deploy it, so it joins the self-serve stack instead of living in someone's editor.

Instrumentation and event-schema design upstream

You don't only analyze what's logged - you shape what gets logged. If the event schema doesn't carry model version, deploy id and a stable run id, no downstream cleverness recovers them. Designing instrumentation is the most impactful tooling work, because a missing field is a regression you can never attribute.

• Insist on a stable run_id and event ordering so sessionization is possible at all.
• Carry the triage dimensions (model version, deploy/commit, region, client version, surface) on the events themselves.
• Log the censoring signal - start, heartbeat, terminal - so aborted runs are distinguishable from fast ones.
• Treat schema as an interface: changes are versioned and announced, not silent.

Data observability so trust doesn't erode silently

A self-serve metric is only as trusted as its freshness. If the pipeline stalls and a dashboard quietly shows yesterday's data as today's, engineers will make a bad call and then stop trusting every dashboard. Observability on the data itself is what keeps the tooling credible.

Detection and clean stats are worth nothing if the finding sits in a notebook while users churn. Cursor runs at high pace, so the last mile - turning a signal into a decision someone acts on today - is where this role earns its seat.

Package findings for people who will read three sentences. Lead with the decision and the recommended action, state your confidence, then put the evidence underneath for whoever wants to verify. The inverted pyramid beats the academic build-up in a fast org.

Quantify blast radius to prioritize

When two regressions compete for one engineer's day, blast radiusHow much breaks if a change goes wrong; the scope of potential damage. decides. Affected users times severity gives a comparable number, so a 0.3% latency bump on the most-used surface can outrank a hard failure that hits a handful of people. Prioritizing by blast radius is how you focus a small team on the frustration that actually dominates.

Escalate vs file - calibrate to impact

Close the loop and write the narrative

A fix is not done when it ships; it's done when the metric confirms the regression resolved. Verify in the same series you detected it, on the same segments and watch for the regression hiding in one slice while the global line recovers. Then write the postmortem-grade narrative so the next person inherits the lesson, not just the patch.

1. 1Verify the fix. Confirm the metric returned to baseline in the affected segments, not just globally.
2. 2Check for displacement. Make sure you fixed the regression rather than moving it to another slice.
3. 3Quantify recovery. Restate blast radiusHow much breaks if a change goes wrong; the scope of potential damage. post-fix so the win is measured, not assumed.
4. 4Write it up. What broke, how it was detected, root cause and the prevention that stops a repeat.