Capstone: Mock Loop & Self-Exam

You don't find out whether your SQL holds up under a clock by reading query patterns. You find out by setting a 60-minute timer, opening a blank editor and talking through every metric definition as you type it. This section is that drill, built on the exact data shape Cursor's screen would hand you: a raw interaction events table.

The DS technical screen is data-shaped, not algorithmic. Assume one fat events table - one row per user↔AI interaction - with columns you'll reference all hour: event_id, user_id, request_id, model, client_version, event_type, latency_ms, status, ts. Set the timer before you read the prompts.

Task 1 - p95/p99 latency by model and client versionthe warm-up; do not average

The trap is reaching for AVG(latency_ms). Latency is heavy-tailed, so the mean is dragged by the tail and hides the slow experiences that actually frustrate users. Report percentiles, segmented.

SELECT
  model,
  client_version,
  COUNT(*)                                              AS n,
  PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY latency_ms) AS p50_ms,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95_ms,
  PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY latency_ms) AS p99_ms
FROM interaction_events
WHERE event_type = 'success'           -- only completed interactions have a real latency
  AND latency_ms IS NOT NULL
  AND ts >= NOW() - INTERVAL '7 days'
GROUP BY model, client_version
HAVING COUNT(*) >= 200                  -- suppress percentiles on thin segments
ORDER BY p99_ms DESC;

Task 2 - the prompt→success funnel with drop-off

Build the stage funnel prompt → retrieval → tool_call → edit_apply → success and show where attempts die. Anchor every stage to request_id so you're counting logical attempts, not raw rows.

WITH per_request AS (
  SELECT
    request_id,
    MAX(event_type = 'prompt')     AS hit_prompt,
    MAX(event_type = 'retrieval')  AS hit_retrieval,
    MAX(event_type = 'tool_call')  AS hit_tool,
    MAX(event_type = 'edit_apply') AS hit_edit,
    MAX(event_type = 'success')    AS hit_success
  FROM interaction_events
  GROUP BY request_id
)
SELECT
  COUNT(*) FILTER (WHERE hit_prompt)    AS s1_prompt,
  COUNT(*) FILTER (WHERE hit_retrieval) AS s2_retrieval,
  COUNT(*) FILTER (WHERE hit_tool)      AS s3_tool_call,
  COUNT(*) FILTER (WHERE hit_edit)      AS s4_edit_apply,
  COUNT(*) FILTER (WHERE hit_success)   AS s5_success,
  ROUND(100.0 * COUNT(*) FILTER (WHERE hit_success)
              / NULLIF(COUNT(*) FILTER (WHERE hit_prompt), 0), 1) AS overall_pct
FROM per_request;

Task 3 - dedup retries into logical attempts before a failure rate

A flaky network produces three rows for one user intent: two timeouts then an ok. Counting raw rows reports a 67% failure rate for something the user experienced as a success. Roll up to the logical attempt first.

1. 1Define the logical attempt. Group retries under one key - request_id or a window of user_id + same prompt within N seconds if no request_id exists.
2. 2Collapse status with a precedence rule. A logical attempt succeeds if any of its tries succeeded; it fails only if every try failed. Decide and state the rule.
3. 3Compute the rate on attempts, not rows. Failure rate = failed logical attempts / total logical attempts.
4. 4Sanity-check the denominator. Confirm attempt count is below row count by roughly your retry rate; if they're equal, your dedup didn't fire.

WITH attempt AS (
  SELECT
    request_id,
    BOOL_OR(status = 'ok')                              AS any_ok,
    COUNT(*)                                            AS tries
  FROM interaction_events
  WHERE event_type IN ('success', 'error', 'timeout')
  GROUP BY request_id
)
SELECT
  COUNT(*)                                              AS logical_attempts,
  COUNT(*) FILTER (WHERE NOT any_ok)                     AS failed_attempts,
  ROUND(100.0 * COUNT(*) FILTER (WHERE NOT any_ok)
              / COUNT(*), 2)                             AS failure_rate_pct,
  ROUND(AVG(tries), 2)                                  AS avg_tries
FROM attempt;

Self-score the run

“Define how we measure agent reliability for Cursor.” That's the prompt. It's deliberately open and the round grades whether you can turn an abstract word - reliability - into something a team could query tomorrow and gate a release on.

Run a fixed structure so the ambiguity doesn't sink you. State what you're measuring and for whom, propose the metric tree, then describe how it lives in the org. Reserve the last third for operationalization, because that's where this round separates a definition from a slide.

1. 1Scope it (60s). “Reliability” for the agent harness means: does an agent run complete, correctly and fast enough that the user keeps going? Name what you're not measuring (model quality of the underlying LLM is a separate pillar).
2. 2Pick the north-star. One metric that captures the experience and moves with real wins. Then the guardrails that must not degrade while you chase it and the counter-metrics that catch you gaming it.
3. 3Make each metric queryable-tomorrow. Give the formula, the unit (logical attempt), the window and the exact event fields.
4. 4Operationalize it. Dashboard, alert, release gate and the ritual that forces a decision.

The metric tree, made concrete

Confront the three things that make this hard

Operationalize it - the part most candidates skip

This round is adversarial by design. You'll propose an A/B, then the interviewer changes the rules - “you can't randomize this one” - and probes every assumption. The grade is method-fit and intellectual honesty, not whether you've memorized a test.

Part 1 - design the A/B for a retrieval changesay all six pieces before they ask

1. 1Unit. Randomize at the user level, not the request level - requests within a user are correlated and request-level assignment leaks the treatment across a session.
2. 2Primary metric + guardrails. Primary: agent success rate. Guardrails: p95 latency and cost per attempt, so a retrieval win that doubles latency or spend doesn't ship.
3. 3MDE and sample size. Decide the smallest success-rate lift worth shipping (say +1pp), then size for it given baseline rate and variance - not the other way around.
4. 4Duration. Run at least one full weekly cycle to cover weekday/weekend mix and long enough to clear novelty.
5. 5Sanity checks. Pre-registered: sample-ratio check, metric of a known A/A invariant and a guardrail tripwire.
6. 6Analysis plan. Fixed in advance - one primary metric, one look (or a sequential procedure) and the decision rule written down before data lands.

Part 2 - “we can't randomize this model swap”

A model rollout often can't be A/B'd: it ships to everyone on a date or routing constraints prevent clean assignment. Now you need a quasi-experiment and the move is matching the method to the structure of the data you actually have.

Part 3 - the curveballs they'll throw

The take-home is the round where craft shows. The brief: “p99 completion latency jumped after release N - find the cause.” You get a synthetic dataset and 4 to 8 hours. What's graded isn't just whether you find it, but whether your write-up reads like something an engineering lead could act on in five minutes.

Diagnose like an incident, not a data dump. Localize, rule out confounds, quantify the blast radiusHow much breaks if a change goes wrong; the scope of potential damage., then lead the write-up with the decision. Most candidates invert that order and bury the recommendation under their EDA.

The diagnosis sequence

1. 1Confirm the jump is real. Plot p99 by day around release N. Is it a step change at the release boundary or drift that started earlier? A change-point on the series beats eyeballing.
2. 2Segment to localize. Cut p99 by model, region and client_version. A global jump that's actually one region or one model narrows the cause fast - and watch for Simpson's paradox in the pooled view.
3. 3Rule out confounds. Did traffic mix shift? A heavier prompt type or a big new cohort can move p99 without any code regression. Hold mix constant and re-check.
4. 4Quantify the blast radiusHow much breaks if a change goes wrong; the scope of potential damage.. How many users, which segments, how much extra latency - translate the stat into impact an eng lead feels.
5. 5Recommend. Roll back, fix forward or investigate further - and say which, with the evidence that decides it.

Structure the write-up - decision first, evidence in the appendix

Self-score with a hiring-bar rubric

The product/craft round is the one Cursor uses to detect superficial usage and for this role it's where reliability instinct meets real product sense. You can't fake it from the docs. You earn it by running Cursor and a rival agent hard, then forming a sharp opinion about where each one breaks.

Record a 5-minute reliability critique, Cursor vs. one competitor

Pick one competing agent you've actually used. Record yourself, on a clock, comparing where reliability holds and where it cracks. Ground every claim in a moment you lived, not a feature list.

1. 1Name the failure you hit. “On a large multi-file refactor, the agent loop stalled mid-tool-call and I had to restart.” Specific, reproducible, dated.
2. 2Compare honestly. Where the competitor is more reliable, say so. Pretending Cursor wins everywhere reads as a pitch, not a critique.
3. 3Tie each critique to a metric. Don't just complain - name the metric you'd ship to catch it.
4. 4Land an opinion. Which reliability problem would you attack first if you had the seat and why that one.

Deliver your 90-second “why this pillar, why me”

• Why reliability over growth: you're drawn to making a non-deterministic product trustworthy at billions of interactions, where defining 'good' is itself unsolved.
• Your proof: a time you operationalized a north-star - a metric that caused a rollback, a launch hold or a reprioritized fix, not just a dashboard you built.
• Why you, specifically: the blend of experimentation/causal rigor with builder instinct and that you work across the stack instead of living in a notebook.
• Why now: an early seat means you write the playbook and you want that ambiguity rather than inheriting someone else's metrics.

Run the readiness checklist across all six modules