1 min lesson
Why evals are essential and genuinely hard
Rebuild the main list in "Why evals are essential and genuinely hard", then say what each item changes.
Step 1 of 3
The practical rule: don't ask "which model is best" in the abstract, ask "best at what." Use a high-reasoning model to produce the plan, then hand the implementation to a fast executor. When you don't want to choose, Auto modeCursor's automatic model router with Cost, Balance and Intelligence options that trade price against model capability. Press Enter for the full definition. routes the model per task. This pairs with the failure-mode work above - a wrong-tier choice (a fast executor asked to do deep planning) shows up as the over-eager or runaway behavior in the table.
Learn more
Full explanation
CursorBench: a real eval to anchor the conversation
CursorBench: a real eval to anchor the conversationperformance AND token efficiency
When the eval discussion gets abstract, ground it in a named, real benchmark: Cursor built its own, CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition., which tests performance and token efficiency together - reflecting the market's shift toward price and efficiency, not just raw capability. What to teach is the dual axis, not a leaderboard. Frontier reasoning models tend to lead on raw performance while sitting mid-to-high on token spend; coding-specific executors (the Codex-style variants) tend to lead on token efficiency at slightly lower performance. The specific rankings churn model to model, so cite the shape of the trade-off, not a frozen version number.
A single-axis benchmark ("who scores highest") hides the cost of getting there - a model that one-shots a feature while burning a billion tokens isn't efficient, and a cheap model that needs 100 re-prompts isn't either. CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition.'s performance-and-efficiency framing is the institutional version of the true-efficiency mindset: optimize for the right tokens, not the fewest. Citing it lets you choose models with data instead of vibes, which is exactly the credible posture the DX role rewards.
When you publish 'here's where the agent struggles and here's how to work with it,' you're not admitting weakness - you're handing developers a mental model that makes them more effective. That post outperforms a glossy one because it's true and the next thing you publish inherits the trust you just built. This is the DX version of truth-seeking: credibility compounds.
Turn a failure into an artifact
The most useful DX move is to take a known failure mode and ship the workaround as content. A 'lost-in-the-middle' bug becomes a short post on structuring long prompts. A runaway loop becomes a guide on stop conditions and verification.
- 1Reproduce it. Get a clean, minimal case of the failure that a reader can run.
- 2Explain the cause. Tie it back to a real mechanism - context limits, retrieval, tool design - not vague 'AI is unreliable.'
- 3Show the fix. Give the exact prompt, rule or tool change that works and show the before/after.
- 4Ship it. A post or a doc that turns a papercut into a repeatable technique is content that spreads because it's useful.
When an interviewer asks 'where do coding agents fall short,' don't hedge and don't trash the product. Name two or three specific modes from the table, explain the mechanism behind each and end with the workaround you'd teach. That arc - honest, mechanistic, actionable - is exactly the voice Cursor wants pointed at the developer community.
QWhy are evals for open-ended coding tasks fundamentally harder than, say, scoring a multiple-choice benchmark?