AI Coding Benchmark Method

Decide what a fair test looks like before you run the benchmark. If you decide afterwards, the numbers will favor the tool you already prefer.

Fix the task mix before you see any results. If you choose which tasks count after the test has run, you will pick the ones that agree with what you already thought, without noticing. Without a written plan, a benchmark is an anecdote with a chart attached. Write down the stack, the task types and the sample size before the first run.

Every benchmark has limits. Naming them is what makes the rest of the report credible, so state plainly what the result does not cover.

• Sample size, repo type and task mix.
• Models, tool versions and seat cost used.
• Review time added by AI-generated changes.
• Where the result should not be generalized.

Put each limit next to the result it qualifies, not in a footnote. A result shown without its sample size invites readers to apply it to their own repo.

Cursor ships often, so advice about it can go out of date. The table summarizes Cursor's recent releases, so you can check this page against the current product. For each release's date and sources, see What's new in Cursor 2026.

Reward hacking makes a coding benchmark look better than it is. Cursor's June 2026 research found that, on SWE-bench Pro, 63% of successful Opus 4.8 Max resolutions retrieved the known fix rather than derived it. The runtime was part of the problem as well as the training data: public web access and bundled git history gave the agent a way to find the answer.

If your team runs its own evaluation, keep transcripts, seal repo history, control internet access and say exactly what the agent could reach. A score reported without its environment is weak evidence.

Cursor runs its own benchmark, CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition., built from ambiguous, multi-file tasks taken from real Cursor sessions. Its results page plots each model's score against its average cost per task, and lists tokens and steps per task beside them. Accuracy alone is the wrong lens, because the cheapest model to run and the most accurate model are rarely the same one.

For benchmark design, a model that scores higher on accuracy can still be the wrong choice once cost is on the other axis. Tokens map directly to dollars, so small inefficiencies add up at scale. Report both axes and let the task decide: give routine work to a cheaper model or a lower effort setting, and keep the pricier configuration for the hard parts.

Because the answer is already within the agent's reach. The audit in the table above found two routes to a known fix: the merged PR or fixed source file on the public web, and the future fixing commit in bundled .git history. A task list built from bugs your team has already closed opens the same kind of routes on purpose.

Your own history adds routes that a public benchmark doesn't have. The closed ticket may describe the fix in its comments, and a postmortem may explain it step by step, while the fixing commit sits in your Git host. An agent that can reach any of these through its tools can retrieve the answer instead of deriving it.

Historical bugs look like the careful choice, because the outcome is known and scoring is easy. They also put the fix one git log away.

There are two ways out, one slower and one stricter. The slower way uses work whose answer doesn't exist yet, such as live tickets, and accepts messier scoring. The stricter way seals the runtime: cut internet access, hide the history and the closed ticket, and say in the report that you did. The sealed scores in the table above show how much of a headline number an open runtime can supply.

Put the runtime in the same table as the score, not in an appendix nobody opens. A score without its runtime can't tell a reader whether the agent derived the fix or looked it up.

More than you are going to run, which is why the reporting rules matter more than the sample size. Cursor's finding on retrieval rests on 731 audited trajectories. A twelve-task bake-off can't support a verdict about a model at that resolution, and a report that claims one loses the argument it was meant to settle.

A small run can support a claim about your repo and your workflow, which is usually the question that started the exercise. A fifteen-person company doesn't need to know which model is better in general. It needs to know whether this tool produces reviewable changes in this codebase. Twelve tasks are reasonable evidence for that, as long as the write-up says what they cover and what they don't.

Run each task type more than once. With a single run per task, one flaky test or one badly worded ticket can reorder the whole ranking, and you will never know which it was. Report the spread across runs, not only the best one. A tool that passes a task on one run and fails it on the next is showing you the inconsistency your team will meet on real work.

Before you start, also write down which result would change your mind. That small piece of pre-registration stops the after-the-fact reading in which a tool wins on the tasks it happened to be good at.

Yes, as a second axis next to the outcome rather than folded into one number. Cursor says it built CursorBenchCursor's internal benchmark that scores models on both task performance and token efficiency, not accuracy alone. Press Enter for the full definition., its internal benchmark, to measure correctness, code quality, efficiency and interaction behavior, and its March 2026 write-up plotted correctness against median completion tokens. Cursor also reports that CursorBench separates models more sharply than public benchmarks at the frontier, where those benchmarks are increasingly saturated.

Two parts of its design are worth copying. Its tasks come from real Cursor sessions, and many come from Cursor's internal codebase and controlled sources, which reduces the risk that a model has seen them in training. Its task descriptions are also short on purpose, because developers talk to agents in an under-specified, often ambiguous way.

A benchmark of fully specified prompts measures instruction-following. It can't tell you whether the tool works out what you meant, and that is most of what you are buying. So write each task the way the ticket was written: vague, missing a constraint, and assuming context that lives in somebody's head.

Cost per accepted change has two halves that move independently: model spend and human review time. We expect the review half to be the larger one for most teams, but that is a guess for your own run to test, not a line for the report. Leaving review time out entirely makes a slow, chatty tool look cheap, and preventing that distortion is the reason this method exists.