Evals & Production Reliability

Every FDE loop bends toward one question: how do you know it works? For most roles that's a follow-up. For this one it's the whole interview wearing a different hat.

A Forward Deployed Engineer at Cursor ships a code-modifying agent into a customer's repo and then has to convince a skeptical senior engineer to trust it on their production code. That engineer has seen confident demos collapse the moment they touch a real codebase. The only thing that moves them is a number they can check.

Hand-waving here is the most common way strong candidates lose offers. They build something that works in the demo, present it well and then go quiet when asked to prove the success rate. The interviewer doesn't hear modesty. They hear a person who can't tell a working system from a lucky one.

Evals make “done” definablethe operational version of a success metric

An FDE who can't define success criteria can't define done. "The migration script works" is not a finish line. "The script converts 412 of our 430 components, the other 18 are flagged for manual review and zero introduce a type error" is a finish line a customer can sign off on.

Cursor deploys an AI coding agent, so eval rigor isn't a hygiene topic bolted onto the role. It's the literal deliverable. When the product you ship is a thing that edits code, "how do you measure that it edited the code correctly" is the job description compressed into one sentence.

An eval set is a list of tasks where you already know the right answer, plus a grader that can decide automatically whether the agent got there. For code, you're lucky: the compiler and the test suite are graders that don't get tired.

Start by collecting golden tasks - concrete inputs with a known-good outcome. For a refactor workflow a golden task might be a real file and the assertion "after the edit this still compiles, the existing tests pass and the public behavior is unchanged." You don't need a thousand of these. You need fifteen that cover the cases you actually care about.

1. 1Pull real examples. Mine the customer's repo and git history for representative tasks, not toy snippets. Adversarial cases live in their codebase already.
2. 2Pin the known-good outcome. Write down what “correct” means per task: compiles, tests pass, type-checks, behavior preserved, no new lint errors.
3. 3Pick a grader per task. Choose the cheapest grader that actually catches the failure you care about.
4. 4Set a gate. Decide the pass rate below which a rollout is blocked and stick to it under deadline pressure.
5. 5Run it on every change. If it's slow or expensive, you'll stop running it and an eval you don't run is a comment.

Choosing graders - and knowing how each one liesevery grader has a failure mode

golden = {
    "id": "refactor-auth-001",
    "input_file": "src/auth/session.py",
    "instruction": "Replace the deprecated session API with the v2 client.",
    "expect": {"compiles": True, "tests_pass": True, "behavior": "preserved"},
}

def grade(task, agent_output):
    if not type_checks(agent_output):        # cheap, deterministic
        return Fail("type_error")
    if not run_tests(task.tests):            # behavior preservation
        return Fail("test_regression")
    verdict = llm_judge(task, agent_output)  # only for taste
    return verdict  # Pass / Fail + reason

Cover the cases that bite, not just the happy path

A demo passes on the file you picked because it looks clean. Production fails on the file with a circular import, the generated code, the 2,000-line module nobody wants to touch. Seed your eval set with the ugly ones on purpose.

• Adversarial inputs: malformed files, partial edits, ambiguous instructions, files that shouldn't be touched at all.
• Edge cases from the real repo: generated code, vendored deps, the one file with no test coverage.
• Negative tests: the agent should sometimes refuse or escalate and the eval should reward that.

What this looks like inside Cursora named benchmark and a DIY review grader

This isn’t abstract. Cursor built its own benchmark, CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone., that scores models on performance and token efficiency together - reflecting the shift toward price/efficiency, not raw capability alone. On it, GPT 5.4 has the best performance while sitting mid-range on token spend; lower-effort GPT 5.3 Codex variants use far fewer tokens at slightly lower performance. The headline efficiency metric is tokens per $100 spent. That dual-axis is the same instinct you want in your own eval set: don’t just ask did it pass, ask what it cost to pass.

You can't debug what you can't see. The first thing to build after the workflow works once is the trace that records what it actually did.

A trace captures one run end to end: the inputs and context that went in, the model's output, the tool calls it made, plus tokens, latency and cost. When a customer says "it did something weird on this PR," the trace is the difference between reproducing it in two minutes and guessing for an afternoon.

What to capture on every model call

The metrics that actually tell you somethingaggregate the traces

Surface failures to a human who can act

A metric that drops is a notification. A useful notification carries the trace link, the failing input, the grader's reason and a guess at the category. The on-call engineer should be able to click once and see exactly what the agent saw.

• Link every alert to the specific trace and the prompt/context that produced it, so root cause is one click away.
• Tag failures by category at capture time (bad context, wrong tool, timeout) so patterns aggregate instead of scattering.
• Keep enough context to reproduce without the customer's help - assume they won't or can't re-run it for you.

You're putting an agent that edits code in front of a team that didn't write it. Roll it out the way you'd roll out a risky migration: quietly first, then a little, then everyone, with a metric gating each step and a way to stop.

Progressive rolloutearn trust one gate at a time

1. 1Shadow mode. The agent runs and produces output, but nothing it does reaches production - you compare its proposals against what humans actually did. Pure signal, zero risk.
2. 2Small cohort. Turn it on for a couple of willing engineers or a low-stakes repo. Watch success rate, cost and their reactions.
3. 3Wider rollout. Expand only when the cohort's metrics hold above the gate. Each expansion is a decision backed by numbers, not a calendar date.
4. 4Full deployment. Default-on for the team, with the eval suite running on a schedule and alerts wired up.

Guardrails - assume the model will sometimes be wrong

Guardrails aren't a lack of confidence in the model. They're the reason a customer lets the model near anything that matters.

Rollback and kill switches

• A kill switch that disables the workflow instantly, owned by someone who can pull it at 2am without a deploy.
• A rollback path for whatever the agent changed - easy when edits go through PRs, which is one more reason to route them that way.
• A documented “what we do when it misbehaves” that the customer's team has actually read, not just received.

Incident response for AIa bad output is an incident

1. 1Triage. Is this one bad PR or is the success rate falling for everyone? Check the dashboard before touching anything.
2. 2Mitigate. Pull the kill switch or narrow the guardrails to stop the bleeding. Mitigation first, root cause second.
3. 3Diagnose. Open the trace, find the variable that changed - model snapshot, context, a prompt edit, an upstream timeout.
4. 4Fix the root cause. Change the prompt, the context, the guardrail or the model, whichever the trace points at.
5. 5Add an eval. Encode the failing case as a golden task so the gate would have caught it. The incident is only closed when it can't silently recur.

Incident response, the other direction: agents as respondersCursor’s own dogfood story

Cursor’s automations feature was born from exactly this problem - it started internally so engineers wouldn’t wake at 3am to comb logs. On a PagerDuty trigger, a cloud agent uses the Datadog MCPModel Context Protocol. A standard that lets an AI agent pull in context from outside the repo, like Jira tickets or internal docs. to fetch logs, investigate, find root cause and - if severity is low - implement a fix. The on-call engineer opens their laptop to a root cause and a PR ready to merge, saving roughly 30 minutes of response time, which matters when an outage can cost millions per minute.

When an agent does something wrong, the amateur reaction is to rewrite the prompt and hope. The professional reaction is to find out which of five things actually broke.

A failure taxonomyname the category before you reach for a fix

The debugging method

1. 1Reproduce. Pin the exact input from the trace. A bug you can't reproduce, you can't fix or verify.
2. 2Inspect the trace. Read what the agent actually saw - the context, the tool calls, the raw output - instead of guessing.
3. 3Isolate the variable. Change one thing at a time. Swap the model, then the context, then the prompt and watch which move fixes it.
4. 4Fix the root cause. Apply the change in the layer the taxonomy points at, not the first layer that's convenient.
5. 5Add an eval. Turn the reproduction into a golden task so the fix is permanent and measured.

Which knob to turnprompt vs. context vs. model vs. guardrails

Tell it as a storythe interview payoff

The behavioral and onsite rounds will ask for a model failure you debugged. The structure that lands is the same every time and it's the structure of the work itself.