1 min lesson
Why evals are essential and genuinely hard
Rebuild the main list in "Why evals are essential and genuinely hard", then say what each item changes.
Step 1 of 2
Why evals are essential and genuinely hard
An eval is how you know whether a change made the agent better or just different. For coding that is harder than it sounds, because the task is open-ended: there are many correct programs, and 'correct' often means 'passes tests a human would accept,' which is expensive to define at scale.
- Open-ended tasks have no single right answer, so exact-match scoring fails and you need behavioral checks such as whether it compiles, whether the tests pass and whether a grader prefers it.
- Real codebases are messy and stateful; a benchmark that runs in a clean sandbox may not predict behavior in a million-line repo.
- Good evals are expensive to build and maintain, and a stale or contaminated eval quietly lies to you about progress.
Learn more
Full explanation
CursorBench: a real eval to anchor the conversation
CursorBench: a real eval to anchor the conversationperformance AND token efficiency
When the eval discussion gets abstract, ground it in a named, real benchmark: Cursor built its own, CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition., which tests performance and token efficiency together, reflecting the market's shift toward price and efficiency as well as raw capability. What to teach is the dual axis, not a leaderboard. Frontier reasoning models tend to lead on raw performance while sitting mid-to-high on token spend. Coding-specific executors, the Codex-style variants, tend to lead on token efficiency at slightly lower performance. The specific rankings churn model to model, so cite the shape of the trade-off, not a frozen version number.
A single-axis benchmark ("who scores highest") hides the cost of getting there. A model that one-shots a feature while burning a billion tokens isn't efficient, and a cheap model that needs 100 re-prompts isn't either. CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition.'s performance-and-efficiency framing is the institutional version of the true-efficiency mindset: optimize for the right tokens. Citing it lets you choose models with data instead of vibes.
Publishing 'here's where the agent struggles and here's how to work with it' hands developers a mental model that makes them more effective. That post tends to land better than a glossy one because it's true, and the next thing you publish inherits the trust you just built. This is the DX version of truth-seeking: credibility compounds.
Turn a failure into an artifact
The most useful DX move is to take a known failure mode and ship the workaround as content. A 'lost-in-the-middle' bug becomes a short post on structuring long prompts. A runaway loop becomes a guide on stop conditions and verification.
- 1Reproduce it. Get a clean, minimal case of the failure that a reader can run.
- 2Explain the cause. Tie it back to a real mechanism: context limits, retrieval or tool design.
- 3Show the fix. Give the exact prompt, rule or tool change that works and show the before/after.
- 4Ship it. A post or a doc that turns a papercut into a repeatable technique is content that spreads because it's useful.
When an interviewer asks 'where do coding agents fall short,' skip the hedging and the product-bashing. Name two or three specific modes from the table, explain the mechanism behind each and end with the workaround you'd teach. That arc, honest then mechanistic then actionable, is the voice to aim for with the developer community.
QWhy are evals for open-ended coding tasks fundamentally harder than, say, scoring a multiple-choice benchmark?