Measuring AI productivity: turning 'we feel faster' into numbers

When licence renewal comes around, two sentences meet in the same meeting. The team says "we are much faster", and the person signing the budget asks "by how much". Somewhere between them sits a screenshot from the vendor dashboard: share of suggestions accepted, lines suggested, active users. That screenshot does not answer the question, because the question is about delivery and the dashboard counts what happens in the editor.

This article turns "we feel faster" into numbers a tech lead can extract from their own Git host within two weeks. Our running example is a fictional team: a retail app called Sepet, fifteen developers spread across iOS, Android and backend, on a two-week release train. Every Sepet number below is made up and exists only to show the shape of a table.

Why the feeling and the measurement drift apart

The most instructive study we have read on this is METR's randomised trial from 2025. Sixteen experienced open-source developers worked on 246 issues from their own repositories, and each issue was randomly assigned to "AI allowed" or "AI not allowed". The result: with AI allowed, issues took 19% longer. The perception side is the interesting part. Before the study the developers expected a 24% speedup, and after living through the slowdown they still believed AI had made them 20% faster.

Reading this as "AI slows you down" would be wrong, and the authors say so themselves: the study covers experienced people in large repositories they know very well, with early-2025 tools. METR's February 2026 update makes the picture more interesting still. When they tried to rerun the experiment with newer tools, noticeably more developers declined to take part because they did not want to work without AI, and 30% to 50% of those who did take part said they held back some tasks they did not want to do without it. The authors think today's speedup is probably higher, and in their own words they treat that data as very weak evidence.

The lesson we take from the two posts is independent of the speed figure. An experienced developer's estimate of their own speed turned out to be too unreliable to stand in for measurement, and running a clean experiment is hard even for people who do it for a living. Do not assume it will be easy in your fifteen-person team.

There are three familiar reasons the feeling drifts from the measurement.

The work moves from writing to reading. Staring at an empty file is tiring, and fixing a draft is less tiring. The developer remembers the reduced effort as reduced time. The time has moved elsewhere: writing the prompt, waiting for output, reading the output, finding the part that looks plausible and is wrong, and into the calendar of the senior who reviews the PR.

There is a novelty effect. The first weeks with a tool are not typical, for the team or for the measurement. People try it on everything, lose time on tasks it does not suit, then learn where it helps. A number taken from the first month misleads in both directions.

Vendor dashboards answer a different question. GitHub Copilot's usage metrics documentation defines acceptance rate as the percentage of suggestions accepted by users, and counts lines suggested and lines added separately. Those numbers answer "is the tool being used", and they are useful for spotting licences nobody opens. They do not tell you whether the accepted suggestion was deleted half an hour later, survived review, or reached a user.

Four metrics that mislead

The Sepet team's first quarterly report looked like this. You will recognise it:

AI pilot results, Q2
- Acceptance rate: 31%
- Lines suggested by AI: 48,000
- PRs per week: 38 → 52
- Sprint velocity: 61 → 74 story points
Conclusion: the team is roughly 35% more productive.

All four lines may have been measured correctly, and the conclusion still does not follow from them.

Lines of code is an output measure, and lines are the cheapest thing AI produces. Between a PR that solves the problem in 40 lines and one that needs 400, the second looks more "productive" while costing more to maintain. We look at line counts only as PR size, to explain review load.

PR count loses its meaning when the definition of a PR changes. Agents split work into small pieces, and small pieces make many PRs. The count can rise while the number of features delivered stays flat.

Acceptance rate is a usage metric. A high value is not always good news either: a team that accepts suggestions without reading them will max it out.

Story points are the team's own estimation unit, and the team estimates knowing the tool exists. A rise in velocity mostly shows that points have inflated. And the day you turn story points into a target, they stop measuring anything.

Everything counted in the editor measures usage. Productivity is measured where the change reaches the user and stays there.

Metrics that hold up, and what each says about AI

The sturdiest framework available is DORA's delivery metrics. The current guide defines five, and two sit at the centre for us: change lead time (the time for a change to go from committed to version control to deployed in production) and change fail rate (the ratio of deployments that require immediate intervention). We add a few that help isolate what AI is doing.

Reporting lead time as a single number is useless, because AI touches only one of its stages. Split it into five: coding (first commit to ready for review), waiting for review, review and fix rounds, CI, and release (merge to the version users get). A pattern we see often in large codebases: the coding stage shrinks, waiting for review grows, the total does not move. What that pattern says is that coding is no longer the bottleneck. On mobile the release stage needs particular attention: for a team on a two-week release train, halving coding time barely changes how soon a user sees the change.

Change failure rate and rework are the bill for speed. We keep them separate because they catch different things: failure rate counts the accident that reached users, rework counts the mistake the team fixed before it did. The typical failure of AI output, "it compiles, it passes the tests, and it does the wrong thing", shows up in rework first.

Review load per senior is the number most often neglected in AI rollouts. In a fifteen-person team most reviews go through three or four people. When the writing side speeds up, those three or four people's weeks fill, reviews get shallower, and nobody sees it as a metric. Track weekly review count and lines reviewed by role, not by person. We cover ways to reduce that load in our article on AI code review.

Bug diagnosis time is where AI often helps most, quietly: reading logs, writing the test that reproduces a crash, finding the suspect commit. It does not show in feature lead time, so measure it separately. The same applies to onboarding time, though in a team that hires two or three people a year it will never be a statistic, only an observation worth writing down.

Getting the data without buying a platform

Engineering analytics platforms give you most of these numbers out of the box. For a fifteen-person team we do not recommend buying one to take the first measurement: learn which question you are asking first, then choose the platform. The data you need is already in your Git host and your repository.

Tag first: marking AI involvement honestly

For segmentation, a three-value label on every PR is enough: ai:none, ai:assisted (written with completion or chat, the developer drove), ai:agent (an agent produced most of the change, the developer reviewed it). A small check that makes the label mandatory stops people forgetting:

# .github/workflows/ai-label.yml
name: ai-label
on:
  pull_request:
    types: [opened, reopened, labeled, unlabeled, synchronize, ready_for_review]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - env:
          LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
        run: |
          count=$(echo "$LABELS" | jq '[.[] | select(startswith("ai:"))] | length')
          if [ "$count" -ne 1 ]; then
            echo "::error::Pick exactly one ai:* label (ai:none, ai:assisted, ai:agent)."
            exit 1
          fi

Where PR labels are not available, a commit trailer carries the same information. Git supports this directly, and git log can read a trailer by key:

git commit -m "Apply campaign discount to basket total" --trailer "AI-Assisted: agent"

# Reading: the trailer value for every commit
git log --first-parent --format='%h %(trailers:key=AI-Assisted,valueonly,separator=%x2C) %s' main

There is precedent: the Linux kernel documentation defines an Assisted-by trailer for contributions written with AI help and states plainly that responsibility stays with the human. We prefer recording the level of involvement over the tool name, because that is what helps in segmentation. If you squash-merge, check once in your own merge settings whether the trailer makes it into the squash commit. If you are not sure, treat the PR label as the source of truth.

Whether the label stays honest depends on two conditions. First, it never feeds into individual evaluation, and you say so in writing. In a team where picking ai:none reads as falling behind and picking ai:agent reads as lazy, the label is meaningless within a month. Second, reports are never produced per person. Write these rules where the team's other AI rules live, in your AI usage policy.

Pulling PR stage timings from the API

The script below uses two GitHub REST endpoints: the pull request list (created_at, merged_at, labels, user) and each PR's timeline. In the timeline, the committed event gives the commit's author date, ready_for_review gives the moment the PR left draft, and reviewed gives the review's submitted_at and state. It needs only the standard library, and a token with read access to the repository.

#!/usr/bin/env python3
"""Writes stage timings of merged PRs as CSV.
Usage: GITHUB_TOKEN=... python3 pr_timings.py owner/repo 2026-06-01 > prs.csv
"""
import csv, json, os, sys, urllib.request
from datetime import datetime

API = "https://api.github.com"
REPO, SINCE = sys.argv[1], sys.argv[2]
HEADERS = {"Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28"}
if os.environ.get("GITHUB_TOKEN"):
    HEADERS["Authorization"] = f"Bearer {os.environ['GITHUB_TOKEN']}"

def get(url):
    with urllib.request.urlopen(urllib.request.Request(url, headers=HEADERS)) as r:
        return json.load(r)

def pages(path):
    page = 1
    while True:
        sep = "&" if "?" in path else "?"
        batch = get(f"{API}{path}{sep}per_page=100&page={page}")
        yield from batch
        if len(batch) < 100:
            return
        page += 1

def ts(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00")) if value else None

def hours(start, end):
    return round((end - start).total_seconds() / 3600, 1) if start and end else ""

out = csv.writer(sys.stdout)
out.writerow(["pr", "author", "ai", "labels", "merged_at", "coding_h", "wait_review_h",
              "review_h", "total_h", "review_rounds", "reviewers"])

for pr in pages(f"/repos/{REPO}/pulls?state=closed&sort=updated&direction=desc"):
    if pr["updated_at"] < SINCE:
        break                               # sorted by update time, the rest is older
    if not pr["merged_at"] or pr["merged_at"] < SINCE or pr["user"]["type"] == "Bot":
        continue

    first_commit = ready = first_review = None
    rounds, reviewers = 0, set()
    for ev in pages(f"/repos/{REPO}/issues/{pr['number']}/timeline"):
        kind = ev.get("event")
        if kind == "committed":
            when = ts(ev["author"]["date"])
            first_commit = min(first_commit or when, when)
        elif kind == "ready_for_review":
            ready = ts(ev["created_at"])    # last time it left draft
        elif kind == "reviewed":
            user = ev.get("user") or {}
            if user.get("type") != "User" or user.get("login") == pr["user"]["login"]:
                continue                    # bot reviews and the author's own comments do not count
            reviewers.add(user["login"])
            when = ts(ev["submitted_at"])
            first_review = min(first_review or when, when)
            if ev["state"] == "changes_requested":
                rounds += 1

    created, merged = ts(pr["created_at"]), ts(pr["merged_at"])
    ready = ready or created
    if first_review and first_review < ready:
        ready = created                     # PR that was reviewed while still a draft
    labels = [label["name"] for label in pr["labels"]]
    ai = next((name for name in labels if name.startswith("ai:")), "untagged")
    out.writerow([
        pr["number"], pr["user"]["login"], ai, "|".join(labels), pr["merged_at"],
        hours(min(first_commit or ready, ready), ready),   # coding: first commit -> ready for review
        hours(ready, first_review),                        # waiting for review
        hours(first_review, merged),                       # review and fix rounds
        hours(min(first_commit or created, created), merged),
        rounds, "|".join(sorted(reviewers)),
    ])

We ran the script against a public repository and confirmed the fields come back as described. It has limits you should know. The coding stage starts at the first commit, so the thinking and prompting before it is invisible. Amending or rebasing keeps the author date, but on a branch squashed down to one commit the "first commit" carries the date of the final version. Durations are wall-clock and include weekends, which is why we will use medians and percentiles and not means. AI review bots arrive with user.type set to Bot, so they are not counted as the "first review". If they were, your waiting-for-review time would drop to two minutes overnight and you would have a false success story.

On GitLab the equivalent is GET /projects/:id/merge_requests. The response includes created_at, merged_at, draft, labels, and prepared_at, the moment the merge request's preparation steps finished.

Counting rework with git log

Our definition of rework is deliberately crude: a change that landed on main counts as "reworked" if, within N days, a commit whose subject starts with fix, revert or hotfix touches at least one of the files it touched. It assumes squash merges and consistent commit subjects (Conventional Commits, for example).

#!/usr/bin/env python3
"""For every change that landed on main: did a fix/revert hit the same files within N days?
Usage (in the repo root): python3 rework.py 2026-06-01 21 > rework.csv
"""
import re, subprocess, sys
from datetime import datetime, timedelta

SINCE, DAYS = sys.argv[1], int(sys.argv[2])
BRANCH = sys.argv[3] if len(sys.argv) > 3 else "HEAD"
FIX = re.compile(r"^(revert|fix|hotfix)\b|\bhotfix\b", re.IGNORECASE)
IGNORE = re.compile(r"(\.lock$|\.md$|^docs/|\.xcstrings$|/strings\.xml$)")

log = subprocess.run(
    ["git", "log", "--first-parent", "--name-only", f"--since={SINCE}",
     "--format=%x1e%H%x1f%cI%x1f%s%x1f%(trailers:key=AI-Assisted,valueonly,separator=%x2C)%x1f",
     BRANCH],
    capture_output=True, text=True, check=True).stdout

commits = []
for record in log.split("\x1e")[1:]:
    sha, date, subject, ai, files = record.split("\x1f")
    paths = {f for f in files.split("\n") if f and not IGNORE.search(f)}
    when = datetime.fromisoformat(date.replace("Z", "+00:00"))
    commits.append((sha, when, subject, ai.strip() or "untagged", paths))
commits.reverse()                                  # oldest first

newest = commits[-1][1]
print("sha,merged_at,ai,files,reworked,days_to_fix,fix_sha")
for i, (sha, date, subject, ai, paths) in enumerate(commits):
    if not paths or FIX.search(subject):
        continue                                   # a fix is not itself a measured change
    if newest - date < timedelta(days=DAYS):
        continue                                   # observation window not complete, leave it out
    hit = next(((s, d) for s, d, subj, _, p in commits[i + 1:]
                if d - date <= timedelta(days=DAYS) and FIX.search(subj) and p & paths), None)
    print(sha[:10], date.date(), ai, len(paths), int(bool(hit)),
          (hit[1] - date).days if hit else "", hit[0][:10] if hit else "", sep=",")

File-level matching produces false positives: a router file everyone touches gets counted on every fix. Do not trust the absolute value. Look at the difference between two periods measured with the same definition. The line that excludes changes whose observation window is incomplete matters. Without it the rework rate of the last three weeks always looks low, and a measurement taken right after rollout comes out in AI's favour.

Release and CI stages

For merge-to-release time, finding the first tag that contains each commit is enough:

git log --first-parent --since=2026-06-01 --format='%H %cI' main | while read -r sha merged; do
  tag=$(git tag --contains "$sha" --sort=creatordate | head -n 1)
  [ -n "$tag" ] && echo "${sha:0:10},$merged,$tag,$(git log -1 --format=%cI "$tag")"
done > release_lag.csv

The tag date is the date you submitted to the store, which is earlier than the date users got it. If you know your store review and phased rollout times, add them. For CI, the GitHub Actions workflow run list gives run_started_at, updated_at and conclusion for every run:

curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/sepet/app/actions/runs?event=pull_request&status=completed&created=%3E%3D2026-09-01&per_page=100" |
jq -r '[.workflow_runs[] | {name, conclusion,
         min: (((.updated_at | fromdate) - (.run_started_at | fromdate)) / 60)}]
       | group_by(.name)[]
       | "\(.[0].name)\t\(length) runs\t\(map(.min) | add | round) min\t\(map(select(.conclusion == "failure")) | length) failed"'

This is the run's wall-clock duration. For billed minutes, which sum parallel jobs, use your host's billing report. With more than a hundred runs you need to paginate.

Designing the comparison

The baseline is taken before rollout. If the tool is already in use, the good news is that Git history can be read backwards: run the scripts for the three months before the rollout date. The only things you cannot recover retroactively are the label and the survey.

You have two designs to choose from. Comparing the same team before and after is cheap and covers everyone. Its weakness is that everything else that changed in the meantime leaks into the result: new hires, a large refactor, a CI migration, the holiday season. A pilot team against a control team filters out some of that noise, but in a fifteen-person organisation the two "teams" are seven people each, doing different work. The problem METR describes in its update appears here too: the people who want the tool volunteer for the pilot, and the comparison is biased from the start.

For a fifteen-person team our choice is before and after for the whole team, segmented by task type. Comparing ai:agent PRs against ai:none PRs within the same period looks tempting, but developers use AI on the tasks they judge suitable for it. What that comparison measures is the developer's choice of task. Do not use the label for causation. Use it for the question "which task types is the gain concentrated in".

Task-type segmentation is mandatory, because the effect of AI varies widely by task. Four or five buckets are enough: new feature, bug fix, refactor or migration, tests, dependencies and configuration. You get this from a PR label or a title prefix as well.

On how long to wait, our recommendation is this: leave the first four weeks after rollout out of the measurement, then collect at least eight to twelve weeks. Release-bound metrics such as change failure rate need at least four or five release cycles, which on a two-week train is two or three months anyway. Pick a baseline period that is similar on the calendar where you can: comparing the summer holidays with the November campaign season says nothing about the tool.

What the sample size problem looks like at fifteen people

If Sepet merges around forty PRs a week, twelve weeks gives roughly five hundred. That sounds like plenty. Split it by five task types and three AI labels and you have thirty or forty PRs per cell, and PR durations are long-tailed: a single ten-day PR drags the mean. So report the median and the seventy-fifth percentile, and check whether the difference can be told apart from noise. A bootstrap is enough for that and needs no statistics library:

#!/usr/bin/env python3
"""Median difference between two periods (or two groups), with a bootstrap interval.
Usage: python3 compare.py baseline.csv after.csv total_h
"""
import csv, random, statistics, sys

def column(path, name):
    with open(path, newline="") as f:
        return [float(row[name]) for row in csv.DictReader(f) if row[name]]

def p75(values):
    return statistics.quantiles(values, n=4)[2]

before, after = column(sys.argv[1], sys.argv[3]), column(sys.argv[2], sys.argv[3])
random.seed(7)
diffs = sorted(
    statistics.median(random.choices(after, k=len(after)))
    - statistics.median(random.choices(before, k=len(before)))
    for _ in range(5000))
low, high = diffs[int(0.025 * len(diffs))], diffs[int(0.975 * len(diffs))]

print(f"before: n={len(before):4d}  median={statistics.median(before):6.1f}  p75={p75(before):6.1f}")
print(f"after : n={len(after):4d}  median={statistics.median(after):6.1f}  p75={p75(after):6.1f}")
print(f"median difference: {statistics.median(after) - statistics.median(before):+.1f}  "
      f"(95% interval: {low:+.1f} .. {high:+.1f})")
print("interval includes zero: the difference cannot be told apart from noise" if low <= 0 <= high
      else "interval excludes zero: the difference is probably real")

To show what the problem looks like, we ran the script on two randomly generated, long-tailed synthetic data sets (60 and 55 PRs, not from a real team):

before: n=  60  median=  20.5  p75=  39.2
after : n=  55  median=  11.6  p75=  42.7
median difference: -8.9  (95% interval: -15.9 .. +5.6)
interval includes zero: the difference cannot be told apart from noise

The median appears to have dropped from 20 hours to 12, enough to put "lead time down 43%" on a slide. The interval, though, stretches to plus five hours, and the seventy-fifth percentile got worse. In a cell of sixty PRs, differences of this size arise by chance. In a fifteen-person team the honest result usually reads like this: "the difference is clear across all PRs, clear in the bug fix bucket, and there is not enough data in the other buckets". Do not be shy about writing that.

Do not break results down per person. The sample shrinks further, the honesty of the label ends, and what you measure becomes the behaviour of people who know they are being measured.

The survey: short, recurring and specific

The numbers tell you what changed, and they do not tell you why. For that we use an anonymous survey every four weeks that takes two minutes. Questions like "are you happy with AI" are useless, because they measure exactly the perception gap METR documented. Questions should look at the last week and at concrete events:

1. Last week, roughly how many hours did you spend fixing or discarding AI output?
   (0 / under 1 / 1-3 / more than 3)
2. Which task did AI help most with last week? (one sentence)
3. Was there a task last week you started with AI and finished by hand? Why did you stop?
4. In the PRs you reviewed, was there a section you approved without fully understanding it?
   (never / once / more than once)
5. How many times last week were you blocked by a PR of yours waiting for review? (0 / 1-2 / 3 or more)

The fourth question is uncomfortable and gives the most valuable signal: if that share is rising, escaped defects will rise a release or two later. The survey's place is to explain the metrics and to pick what to look at in the next measurement. Do not report a survey result on its own as proof of a gain.

Putting the cost side on the same page

If the gain is measured in hours, bring the cost into the same month and the same unit. There are four items.

Licences and usage: the per-seat fee, plus whatever is billed by usage (tokens, agent runs, premium requests). The second part moves month to month and grows as agent use grows. Do not estimate it, read it off the invoice.

Review time: the review round count and reviewer list in prs.csv give you the number of reviews per week. How many minutes a review took cannot be derived from Git. Ask your seniors to keep a rough log for two weeks, or add a question to the survey.

CI minutes: agents push often, and every push runs the pipeline. Run the query above for the baseline and for the period after. For mobile teams on macOS runners this item can exceed the licences.

Upkeep: rule files, prompts, the agent environment, the labels and the measurement scripts. Small, though not zero, and it usually lands on one person.

Do not convert the gain into "people saved". A shorter lead time removes nobody's salary from the budget. What you put side by side is this: total monthly cost, how many hours were cut and in which stage, and what the quality metrics did.

One page for leadership

Keep the report to one page and always in the same shape. A fictional example for Sepet:

The "Uncertainty" column should not be negotiable. Without it the table turns into a sales brochure.

Sentences we suggest you avoid, and what we write in their place:

One oversold report uses up the credibility of every report after it. The team that writes "40%" in the first quarter has to explain in the second why it is not 80%.

When the numbers do not move

After three months the table may come out flat. That is a valid result and it tells you something. The places we look, in order:

The stage breakdown. If total lead time is flat, coding is shorter and waiting for review is longer, the tool works and the gain is evaporating in a queue. The fix is not on the AI side: review capacity, a PR size limit, automated review as a first pass. If the release stage is most of the total, making coding faster gives users nothing, and the conversation to have is about release frequency.

The task-type breakdown. If the gain is concentrated in the tests and migration buckets and absent from feature work, that is the tool's real profile in your codebase. Steer usage to where it works, and stop recommending it to everyone for everything.

PR size. If median PR size grew after rollout, you are paying the coding gain back in review. Shrink the scope of what you hand to the agent.

Usage itself. This is the one place the vendor dashboard earns its keep: if half the licences are never opened, you are measuring the effect of half a team. Ask the people who do not use it why. The answer is often "it does not work in our module", and that is information about your rule files and project context.

If nothing moves after all of that, narrowing the scope is a legitimate decision: limit licences to the roles that benefit, and measure again a quarter later. We describe how we sequence the whole transition in our article on AI and the SDLC. Measurement is the first step in that sequence.

Checklist

• PR stage timings, rework and CI minutes for the three months before rollout have been extracted and stored.
• Every PR carries exactly one ai:* label and one task-type label, and CI checks it.
• It has been announced in writing that the label and the metrics will not be used in individual evaluation.
• Lead time is reported in five stages: coding, waiting for review, review, CI, release.
• AI review bots are excluded from the "first review" calculation.
• The rework definition is written down, and changes with an incomplete observation window are excluded.
• Weekly review count per senior is tracked.
• The first four weeks after rollout are excluded, and comparison periods are similar on the calendar.
• Median and seventy-fifth percentile are reported, and the bootstrap interval has been checked for every difference.
• Comparisons between labels are not presented as evidence of causation.
• A five-question survey runs every four weeks and is used only to explain the metrics.
• Licence, usage, CI and review cost are on the same page for the same month.
• The leadership report is one page, the uncertainty column is filled in, and there is no "people saved" figure or single headline percentage.
• The four places to look when numbers do not move (stage, task type, PR size, usage) are in the report.