Benchmark Report: AI Coding Tools for Python Services

A benchmark is only as honest as its method. Decide what a fair test looks like before you run it, or the numbers will just flatter the tool you already prefer.

Decide the task mix before you see any results. This sounds like pedantry and isn't: choose which tasks count after the test has run and you'll pick the ones that agree with what you already thought, and you'll do it without ever noticing you did. Writing down the stack, the task types and the sample size in advance is most of what separates a benchmark from an anecdote with a chart attached.

Every benchmark has limits, and naming them is what makes the rest trustworthy. Be explicit about what the result does not cover.

• Sample size, repo type and task mix.
• Models, tool versions and seat cost used.
• Review time added by AI-generated changes.
• Where the result should not be generalized.

Cursor ships often, so here is the current state of the surfaces this page touches. Each row links to the source where you can confirm the detail.

Because in most Python services the test suite is the only automatic verdict available. Type hints are optional and are only checked if something in your pipeline runs a checker, so unless the repo already gates on one, a task scored as "did it implement the feature" has no machine signal underneath it and quietly becomes a vote on reviewer taste.

No machine signal is a shade too strong, I'll admit. The service still has to boot and the linter still runs. Both are weak next to a red test going green, and a benchmark needs something binary at the bottom of it or the scoring drifts week to week.

Failing-test repair gives you that binary. Pull real failures out of your own history where the fix is known and small, then hide the fixing commit before the run.

Hide it properly, too. Out of the branch the container gets, not just out of the prompt.

The part I would watch is how those failures get chosen. Take them all from one module and you have measured one module. Take the easy ones and every tool passes, the report reads like a tie, and you make the decision on price because nothing else separated them.

A second task type earns its place alongside repair: a behaviour change in a service whose suite covers it thinly. That is the honest half of Python work, and it is where you find out whether a tool writes a test for the path it just changed or leaves the coverage exactly where it found it.

More than most reports admit. Cursor's June 2026 research audited 731 Opus 4.8 Max trajectories on SWE-bench Pro and found that 63% of successful resolutions retrieved the known fix rather than derived it: in 57% of audited trajectories the agent located the merged PR or the fixed source file on the public web, and in 9% it mined the bundled .git history for the future fixing commit. With git history sealed and internet access restricted, Opus 4.8 Max fell from 87.1% to 73.0% and Composer 2.5The current Composer release, better at long-running tasks and at judging when a job needs a light touch versus deep work. Press Enter for the full definition. from 74.7% to 54.0%.

That finding is, really, the entire argument for writing a method section at all. The same score moved over 14 points on one model and nearly 21 on the other without anybody touching the model, and the only variable was what the runtime could reach. The benchmark method page carries the rest of that research.

For Python services the runtime question has a second half, which is package installation. If one tool's container can reach a package index and another's cannot, you have benchmarked network policy and printed it as a tool comparison.

So pin the image, pin the lockfile, and record whether the agent could install anything at all. I would also log every install it attempted, successful or not. An attempted install on a task that should have been a two-line fix tells you something about approach even when the resulting patch is correct.

Publish the runtime next to the score, or the score is not comparable to anyone else's and barely comparable to your own from last quarter.

Accepted changes, counted after human review, and nothing else. It sounds obvious written down. The version that ends up in most spreadsheets divides by attempts, because attempts are the number the tooling hands you for free.

Follow what that does to the ranking. A tool that produces four patches for one task, three of them discarded, looks cheaper per unit than a tool that produced one good patch. The reviewer who read all four does not appear anywhere in the arithmetic.

Two cost lines always get counted: usage spend and seats. The line that probably decides the answer is reviewer time, and it is the one nobody instruments.

The rule I would hand someone is to time reviewer minutes per task. Actually a timer is the wrong instrument, because nobody keeps one running honestly for two weeks and the data ends up half-invented. Count review passes instead: how many times a patch came back before it merged. It is coarser, and you will actually have the data at the end of the fortnight.

Include the rejected patches in the numerator while you are at it. Model spend does not refund itself when you throw the work away.

Name it explicitly, because "Python" covers at least two measurement problems that share a language and nothing else. A service with strict typing and a locked dependency set is not evidence about a data pipeline built out of notebook-shaped code and a suite nobody trusts. A result from one says very little about the other.

A two-service team should probably not run this report. Run one task type against two tools on the service you are most nervous about, read the patches, and pick. Four data points published as a percentage is noise with a decimal point attached.

For a platform org I would flip the priority. Environment discipline matters more than sample size, because a hundred tasks in a runtime you never pinned produces a precise wrong answer, and precision is what makes it hard to argue with later.

State the suite's own reliability too. If a slice of your tests fails intermittently, a few points of difference between two tools is your flakes talking, and the fix is to quarantine those tests before the benchmark rather than explain them away in the write-up.

Then say which services the tasks came from, by name, in the report itself. A reader who knows your codebase can discount a result that came entirely from the one service everybody agrees is a mess, and a reader who does not know it deserves the chance to ask. This is the field that costs nothing to include and is missing from nearly every internal benchmark I have been shown.