How to Measure AI Coding Productivity and ROI

Hold two pillars side by side: velocity and quality. Velocity on its own rewards shipping faster even when the code rots. Quality on its own rewards safety even when nothing ships. Track them together and the picture stays honest.

Pillar one: velocity

Velocity has three layers, and they trade ease of measurement against closeness to the outcome you actually care about.

Those rows are ordered by what each one costs to measure, not by how much each one matters. PR velocityHow quickly pull requests are merged; the easiest delivery metric to measure, though it sits furthest from the customer outcome. Press Enter for the full definition. is free because your git host already counts it, and that, honestly, is the whole argument for starting there. Feature-completion velocity needs a shared definition of a completed feature, and somebody has to sit in a room and write that definition down.

Start at the top row and write down the date you intend to stop relying on it. Teams that open at feature-completion because it is the better metric usually spend a quarter defining terms and turn up to the review with nothing plotted. Going the other way is more common, I'd guess. PR velocityHow quickly pull requests are merged; the easiest delivery metric to measure, though it sits furthest from the customer outcome. Press Enter for the full definition. goes on the dashboard, no successor ever gets named, and a year later people are still defending PR counts.

Pillar two: quality

Quality has an external face and an internal one. Watch both, because more code from an agent can quietly trade one for the other.

The two faces move on different clocks. Customer-facing defects respond inside a release or two; coverage and maintainability move over quarters, because they describe the shape of the codebase rather than this week's output. Read them on the same cadence and a good quarter gets scored as a regression.

A small team should measure feature completion and customer-facing defects, and skip everything in between. A large one needs the middle layers, because it has to compare across teams that do not share a definition of anything.

Story points travel worst of the three. They only mean something when the same people estimate the same way twice, which is a lot to ask of a five-person team with no estimation history, and larger orgs get the opposite problem: points exist on every board and mean something different per squad. Roll twelve teams into one story-point number and you have averaged twelve different units.

The other difference, and to me the bigger one, is who reads the number. On a small team the person measuring is also the person shipping, so a rough count checked monthly is enough to act on. Once the number travels to someone who is not in the code, it starts getting compared, and comparison is what forces the definitions. Budget time for that argument before you promise the number.

Both feel measurable, and both mislead. More AI-written code is not better code, and it is not faster delivery. It is just more code to review and maintain.

"Active users" hides depth. Someone who opens the agent once a week counts the same as someone who has rebuilt their workflow around it. A login count tells you reach, which is a smaller thing than adoption and gets mistaken for it constantly.

A seat count worth reporting probably counts the people who used the agent on work they would previously have done by hand.

What goes wrong is duller than it sounds. The number is already sitting there, so it goes on the slide, and by the second quarter the slide is what people get managed against. Once a team is read on how much code the agent wrote, the cheapest way to look better is to let the agent write more of it, and the review queue absorbs the difference.

Code half-life is how long a piece of code survives before it has to change. Short-lived code that gets rewritten within days suggests the first pass missed the mark or the design could not hold weight.

Read it as a maintainability signal. Code that stays stable for a long time is usually code that fit the problem; code that churns constantly is a flag worth investigating, whoever or whatever wrote it.

Check what kind of work a short half-life covers before you read it as a quality problem. A surface nobody has shipped yet churns because the design is still being found, and a config file churns because that is what config files do. Hold the task type steady, watch the same kind of change over two or three months, and the number starts to mean something.

A bigger confound sits underneath, and it is what the code was asked to do. A short half-life on agent-written code reads like a verdict on the agent, and I read it that way for a while. It is closer to a verdict on the request. Vague tickets tend to be the ones people hand to an agent, so a fair amount of what you are measuring is which work got specified properly. That moves the fix upstream, to how the task was written.

Cursor ships its own measurement surface so you do not have to infer everything from git. Four pieces do most of the work.

The point of Cursor BlameAn augmented git blame that records line-level human and agent co-authorship, so you can trace which code was written by AI versus a person. Press Enter for the full definition. is attribution. Once you know which lines came from an agent, you can ask whether agent-authored code carries more defects or churns faster than human-authored code. How much that comparison tells you seems to depend more on the team than on who wrote the line.

Wire up the Analytics API early, for a reason that is mostly political. A number that lives only in the vendor's dashboard gets read as a vendor number. Pull it into the same reporting stack as your delivery metrics and that argument does not come up.

Plan around what this does not record. The audit log covers administrative actions, not prompts or generated code, so the record of what was asked for and what came back is one you build yourself with hooks. There is a fuller treatment of it on how to measure AI developer productivity.

A University of Chicago study, published in November, looked at 1,000 organisations adopting Cursor and found a 39% increase in org-level output.

Take the 39% as one well-sourced data point, not a guarantee for your team. It is an org-level output figure tied to Agent becoming the default, measured across a large sample, which is why it is worth citing exactly as published.

What that study measured was a change of default. It says nothing about any individual getting better at anything, and it does not survive being restated as "39% faster", because output and elapsed time are different quantities. Cite it to argue that the default is worth changing, and stop there.

The acceptance finding underneath it interests me more. Experienced developers accepted more agent-written code, and the planning explanation for that (seniors specify the work first, so what comes back lands nearer to what they asked for) is our inference rather than anything the figure establishes. If it holds, the lever sits upstream of the model, in plan mode and in how the request gets written. Conversation InsightsA Cursor analytics view that passively categorises what agents are doing (new features, bug fixes, refactors) so leaders can see where engineering time goes. Press Enter for the full definition. is pointing at the same thing when it flags an under-specified turn.

Time-to-market acceleration. Are features reaching customers faster than they did before? If the answer is yes and defects are not climbing, the tooling is paying for itself. If you can only watch one number, watch this one.

Everything else can become noise. PR counts and acceptance ratios are useful inputs, but they are means to the end of shipping value sooner, and the failure I would watch for is a team optimising them while the roadmap stalls.

When time-to-marketHow fast a feature actually reaches customers; the outcome metric the velocity and quality pillars ultimately serve. Press Enter for the full definition. does not move, the tooling is rarely the first place to look. Find where a change waits instead: the review queue, or a release train that runs weekly no matter what you do. Faster generation cannot compress a queue, and a change that sits a week waiting on a release slot swallows whatever the agent saved without leaving a mark on any of the numbers above.

Cycle time gets proposed as a substitute for this, and it nearly works. First commit to merge is cheap to pull, and it also moves with review load and scope changes, so a fair share of what it registers has nothing to do with an agent. Keep it for the moments when time-to-marketHow fast a feature actually reaches customers; the outcome metric the velocity and quality pillars ultimately serve. Press Enter for the full definition. surprises you and you need somewhere to start looking.