Benchmark Report: AI Coding Tools for TypeScript Teams

A benchmark is only as honest as its method. Decide what a fair test looks like before you run it, or the numbers will just flatter the tool you already prefer.

Decide the task mix before you see any results. This sounds like pedantry and isn't: choose which tasks count after the test has run and you'll pick the ones that agree with what you already thought, and you'll do it without ever noticing you did. Writing down the stack, the task types and the sample size in advance is most of what separates a benchmark from an anecdote with a chart attached.

Every benchmark has limits, and naming them is what makes the rest trustworthy. Be explicit about what the result does not cover.

• Sample size, repo type and task mix.
• Models, tool versions and seat cost used.
• Review time added by AI-generated changes.
• Where the result should not be generalized.

Cursor ships often, so here is the current state of the surfaces this page touches. Each row links to the source where you can confirm the detail.

Because they fail at different scopes. A test tells you one package still behaves the way you asked. The typecheck tells you the rest of the repo still compiles against the change. Roll them into one pass/fail and you cannot tell which of the two a tool actually cleared.

Here is the case where that bites, and it is a mundane one. A patch lands inside one workspace package, its own tests go green, and a dependent package no longer builds because an exported type quietly narrowed. If the harness runs the package's test command and records a pass, the tool with the widest blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition. scores best, which is the opposite of what anyone wanted to measure.

Run the repo-wide typecheck as its own gate after the test run and record both results per task. The extra cost is one command.

This is, I'd guess, the biggest single reason published TypeScript comparisons disagree with each other. Two teams can run near-identical prompts against near-identical repos and name different winners, because one of them checked the whole project and the other checked a folder. Neither report says which, so neither can be reconciled with the other.

There is a smaller decision inside this one: which typecheck. If CI builds with project references rather than one flat pass, use CI's command. A hand-rolled invocation measures a configuration nobody ships, and it usually measures it more generously.

Record the ordering too. Tests first, then the repo typecheck, and never the reverse, because a patch that breaks compilation will often break the test runner in ways that look like a test failure and get filed as one.

Four types, run in equal numbers. The method above says to fix the task mix before you look at a single result; this is what that mix looks like in a TypeScript repo, with the reason each row is in it.

The fourth row is the one teams leave out, and to me it is the row worth most. A version bump produces many errors with a single underlying cause, which is exactly the shape of work agents are supposed to be good at, and it looks nothing like the small self-contained tasks that usually end up in these reports.

Keep the repo identical across tools and keep the prompts identical too, down to the wording. Rewriting a prompt because one tool "needed more context" is a real finding about that tool, so log it as a finding rather than fixing it in place.

Whether to include a greenfield task at all is genuinely arguable. I would leave it out. What you are buying is help inside code that already exists, and a from-scratch file has no local patterns available to violate, so every tool looks competent on it.

You read the diff for escape hatches. A type error can be resolved or it can be muted, and both produce a green typecheck: an any, a double assertion through unknown, a return type widened until the mismatch disappears, a @ts-expect-error parked on the line that used to fail. Each of those is a pass in the harness and a regression in the repo.

Which is why an automated score cannot carry this report on its own. It just cannot.

The cheap version of the check is a grep over each diff for those four patterns, plus one reviewer line per task. The expensive version is a second reviewer who never learns which tool produced which patch. If you can afford the expensive one, blind the patches, because knowing the vendor changes how generously people read a diff and it changes it in both directions.

The rule I would write down first is never to score a task green on a check the tool could have edited. That is not quite the rule either, though, since the tool can edit the test file as easily as the source. The rule that actually holds is to pin the checks. Write the test and the typecheck command before the run, keep both outside the tool's writable scope, and score against the pinned copy.

Pinning also settles an argument you would otherwise have at the end. When one tool's patch fails a pinned test, somebody will suggest the test was wrong. Sometimes it is. Decide that before you know whose patch it was.

Less than the method implies. With five engineers you will not reach a sample size that supports a percentage, so do not publish one. Run two tools over the same four tasks, read all eight diffs yourself, and write down which ones you would have merged.

That is, I think, a better use of a week than a wider matrix would be, given the decision is small and reversible. You are picking a default for a team you can talk to in one room, and you can change it in a month.

The large-org version fails in the opposite direction. Sample size stops being the constraint and representativeness becomes it: run the benchmark in the pilot team's tidy service repo and you will learn nothing about the monorepo where most of your review time is actually spent. The TypeScript workflow guide covers the package-boundary habits that make that repo the hard case.

The most useful sentence in a large-company report is honestly the one naming which repo the numbers came from. Everything downstream leans on it, and it is the line most often missing.

A caution about scaling this up: ten tasks per tool read by one reviewer will beat a hundred tasks scored by a rota, because the variance lives in the scoring rather than the running.