Benchmark Report: AI Coding Tools for Python Services

Decide what a fair test looks like before you run the benchmark. If you decide afterwards, the numbers will favor the tool you already prefer.

Fix the task mix before you see any results. If you choose which tasks count after the test has run, you will pick the ones that agree with what you already thought, without noticing. Without a written plan, a benchmark is an anecdote with a chart attached. Write down the stack, the task types and the sample size before the first run.

Every benchmark has limits. Naming them is what makes the rest of the report credible, so state plainly what the result does not cover.

• Sample size, repo type and task mix.
• Models, tool versions and seat cost used.
• Review time added by AI-generated changes.
• Where the result should not be generalized.

Put each limit next to the result it qualifies, not in a footnote. A result shown without its sample size invites readers to apply it to their own repo.

Cursor ships often, so advice about it can go out of date. The table summarizes Cursor's recent releases, so you can check this page against the current product. For each release's date and sources, see What's new in Cursor 2026.

Because the test suite may be the only automatic verdict you have. Python type hints are optional, and nothing checks them unless your pipeline runs a type checker. Without that gate, a task scored as "did it implement the feature" has no strong machine signal under it, and the score turns into a vote on reviewer taste.

The service still has to boot, and the linter still runs, but both are weak next to a red test going green. A benchmark needs a binary check at the bottom, or its scoring drifts from week to week.

Failing-test repair gives you that binary check. Pull real failures from your own history where the fix is known and small, then hide the fixing commit before the run. Hide it from the container, not only from the prompt: the branch the agent checks out must not contain the fix.

Watch how you choose those failures. Take them all from one module and you have measured one module. Take only the easy ones and every tool passes, the report reads like a tie, and you end up deciding on price because nothing else separated the tools. Spread the set across modules, and include a few failures that took your own team more than one attempt to fix.

Add a second task type next to repair: a behavior change in a service whose suite covers that path thinly. It shows whether a tool writes a test for the path it just changed or leaves coverage where it was.

By double digits, in Cursor's own test. Cursor's June 2026 research audited 731 Opus 4.8 Max trajectories on SWE-bench Pro and found that 63% of successful resolutions retrieved the known fix rather than derived it. In 57% of audited trajectories, the agent found the merged PR or the fixed source file on the public web. In 9%, it mined the bundled .git history for the future fixing commit.

With git history sealed and internet access restricted, Opus 4.8 Max fell from 87.1% to 73.0%, and Composer 2.5The current Composer release, better at long-running tasks and at judging when a job needs a light touch versus deep work. Press Enter for the full definition. fell from 74.7% to 54.0%. Neither model changed between those runs. Only what the runtime could reach changed, and the scores moved by more than 14 points on one model and nearly 21 on the other. The benchmark method page covers the rest of that research.

Python services add a second runtime question: package installation. If one tool's container can reach a package index and another's cannot, you have benchmarked network policy and printed it as a tool comparison.

So pin the image and the lockfile, and record whether the agent could install anything at all. Log every install it attempted, successful or not. An attempted install on a task that needed a two-line fix tells you something about the tool's approach, even when the final patch is correct.

Publish the runtime next to the score: the image tag, the lockfile, whether the package index was reachable, and whether git history was in the checkout. Without that record, the score can't be compared with anyone else's, and barely with your own from last quarter.

Accepted changes, counted after human review, and nothing else. The tempting denominator is attempts, because the tooling hands you that number for free.

Follow what attempts do to the ranking. A tool that produces four patches for one task, three of them discarded, looks cheaper per patch than a tool that produced one good patch. The reviewer who read all four appears nowhere in that arithmetic.

Usage spend and seats are easy to count. Reviewer time is harder to collect, and it can decide which tool comes out cheaper. Two tools with the same model spend can differ in how often their patches go back for another round, and each round costs reviewer time you are not counting.

So count review passes rather than minutes: how many times a patch went back before it merged. A per-task timer sounds more precise, but it rarely survives two weeks of real work, and the gaps get filled in from memory. Review passes are coarser, and you will still have the data at the end of the two weeks.

Count the rejected patches in the cost as well. Model spend doesn't refund itself when you throw the work away.

Name it, because "Python" covers at least two measurement problems that share a language and little else. A service with strict typing and a locked dependency set is not evidence about a data pipeline built from notebook-shaped code and a suite nobody trusts. A result from one says very little about the other.

A team with two services doesn't need this report. Run one task type against two tools on the service you worry about most, read the patches, and pick. A result that small is too thin to publish as a percentage.

A platform org has the opposite priority. Environment discipline matters more than sample size, because a hundred tasks in a runtime you never pinned produce a precise wrong answer, and the precision makes it hard to argue with later.

State the suite's own reliability too. If some of your tests fail intermittently, a few points between two tools can be flaky-test noise. Quarantine those tests before the benchmark instead of explaining them away in the write-up.

Then name the services the tasks came from, in the report itself. A reader who knows your codebase can discount a result that came entirely from the one service everybody agrees is a mess, and a reader who doesn't know it deserves the chance to ask.