AI Coding Benchmark Method

A benchmark is only as honest as its method. Decide what a fair test looks like before you run it, or the numbers will just flatter the tool you already prefer.

Decide the task mix before you see any results. This sounds like pedantry and isn't: choose which tasks count after the test has run and you'll pick the ones that agree with what you already thought, and you'll do it without ever noticing you did. Writing down the stack, the task types and the sample size in advance is most of what separates a benchmark from an anecdote with a chart attached.

Every benchmark has limits, and naming them is what makes the rest trustworthy. Be explicit about what the result does not cover.

• Sample size, repo type and task mix.
• Models, tool versions and seat cost used.
• Review time added by AI-generated changes.
• Where the result should not be generalized.

Cursor ships often, so here is the current state of the surfaces this page touches. Each row links to the source where you can confirm the detail.

Reward hacking makes a coding benchmark look better than it is. Cursor's June 2026 research found that, on SWE-bench Pro, 63% of successful Opus 4.8 Max resolutions retrieved the known fix rather than derived it. The problem was not only training data. It was the runtime: public web access and bundled git history gave the agent a way to find the answer.

For a team running its own evaluation, the practical move is simple: keep transcripts, seal repo history, control internet access and say exactly what the agent could reach. A score without the environment is not enough evidence.

Cursor runs its own benchmark, CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition., and it deliberately scores two axes together: task performance and token efficiency. Accuracy alone is the wrong lens, because the cheapest-to-run model and the most accurate model are rarely the same one.

The takeaway for benchmark design: a model that scores "better" on accuracy can still be the wrong choice once cost is on the axis. Tokens map directly to dollars, so small inefficiencies compound at scale. Report both axes and let the task decide - match the cheap, coding-specific model to routine work and reserve the pricier high-end model for the hard parts.

Because the answer is already in the repository you handed the agent. Cursor's audit of 731 Opus 4.8 Max trajectories on SWE-bench Pro found the agent locating the merged PR or the fixed source file on the public web in 57% of them, and searching bundled .git history for the future fixing commit in 9%. Build your task list out of bugs your team has already closed and you have opened both of those routes on purpose.

This is the part of a bake-off that feels most responsible and does the most damage. Picking historical bugs is exactly what a careful person does, because the outcome is known and scoring is easy. It is also what puts the fix one git log away.

The way out is either slower or stricter. Slower means work whose answer does not exist yet, so live tickets and a messier scoring process. Stricter means sealing the runtime: cut internet access, hide the history, then say in the report that you did. Cursor's sealed run moved Opus 4.8 Max from 87.1% to 73.0% and Composer 2.5The current Composer release, better at long-running tasks and at judging when a job needs a light touch versus deep work. Press Enter for the full definition. from 74.7% to 54.0%, which is the scale of what an unsealed environment can be contributing to a number you are about to make a purchasing decision on.

Report the runtime alongside the score, in other words. Not as a methodology appendix nobody opens, but in the same table as the result, because the two are not separable claims.

More than you are going to run, which is why the reporting rules matter more than the sample size does. Cursor needed 731 audited trajectories to support one claim about retrieval behaviour. A twelve-task bake-off does not carry a verdict about a model at that resolution, and pretending otherwise is how a benchmark loses the argument it was meant to settle.

What a small run does carry is a claim about your repo and your workflow, which is usually the question that prompted the exercise. Nobody at a fifteen-person company actually needs to know which model is better in general. They need to know whether this tool produces reviewable changes in this codebase, and twelve tasks is a reasonable amount of evidence for that as long as the write-up says so.

Repeat each task type more than once, though. With a single run per task, one flaky test or one badly-worded ticket can reorder the whole ranking, and you will never find out which it was.

The other habit that saves a report is writing down, before you start, what result would change your mind. It is a small piece of pre-registration, and it is what stops the post-hoc reading where a tool wins on the strength of the tasks it happened to be good at. Awkward to write and cheap to do.

Yes, and CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition.'s design is the argument for it. Cursor's internal benchmark is built from real engineer queries, kept out of training sets, and it scores quality and completion tokens together, so a correct-but-bloated answer does not read as well as a correct and tight one. Cursor also reports that it separates strong models from weaker ones more sharply than SWE-bench does.

The design choice worth copying is a different one. CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition. keeps its prompts short and under-specified on purpose, because resolving the ambiguity is part of the task the way it is for an engineer. A benchmark of fully specified prompts measures instruction-following. It cannot tell you whether the tool works out what you meant, and working out what you meant is most of what you are buying.

Write the task the way the ticket was actually written. Vague, missing a constraint, assuming context that lives in somebody's head.

Cost per accepted change carries two halves that move independently: model spend and human review time. My instinct is that the review half is the bigger number for most teams. That is a hypothesis your own run should test, not a line to assert in the report. What is not in doubt is that leaving review time out entirely makes a slow, chatty tool look cheap, and that is the distortion this whole method exists to prevent.