Developers using AI tools took 19% longer to accomplish tasks than developers who didn’t, while believing they had increased their speed by 20–24%. That’s from METR (Model Evaluation & Threat Research). The gap between perception and instrumented reality is the core problem, not noise in the measurement.

The DORA 2025 State of AI-Assisted Software Development report fills out the macro picture: 90% of developers are now using AI coding tools, individual task completion jumped 21%, and pull requests merged increased 98%. Organizational delivery metrics remained completely flat. Faros AI’s telemetry from over 10,000 developers adds the mechanism: as PR volume skyrocketed, code review time increased 91%, PR size grew 154%, and bug rates climbed 9%. Activity rose dramatically while outcomes didn’t move.

PR volume, lines of code, and commit frequency are the nearest available signal. They measure motion well, but progress needs different instrumentation.

Three tiers, three time horizons #

The standard framing (Activity → Productivity → Impact) is correct. The difficulty is that each tier has a different time horizon, and quarterly ROI pressure tends to compress them together.

Activity metrics (TLOC, PR count, commit frequency) are cheap to collect and arrive immediately. They’re useful as leading indicators; disengagement, context-switching overhead, and onboarding friction all surface here early. What they can’t tell you is whether the work was well-directed: a signal worth having, but not the one it’s often asked to provide.

Productivity metrics (features shipped, bugs resolved, roadmap milestones) are more meaningful but lag by weeks or months. AI introduces a specific distortion here: AI-generated code now accounts for roughly 41% of all code written globally, and that code shows 1.7x higher defect rates, 8x more code duplication, and 30–41% increases in technical debt. Shipping volume goes up while foundation quality goes down, and the two don’t cancel each other out.

Impact metrics (revenue, customer NPS, retention) answer “did this matter?” but they’re the hardest to attribute. A developer who ships a well-executed feature into insufficient market demand didn’t fail; the product strategy did. When execution quality and market success are treated as the same signal, the incentive structure breaks and the measurement becomes untrustworthy for both purposes.

The fix is acknowledging each tier’s own time horizon and building instrumentation that holds all three, rather than using activity as a stand-in for impact because it’s easier to report.

The productivity paradox #

The AI Productivity Paradox is well-documented at this point: individual output metrics rise while organizational delivery stays flat or regresses. Individuals speeding up doesn’t automatically speed up the system they belong to.

AI’s ability to generate code quickly leads developers toward larger, more complex PRs, which take longer to review, are harder to reason about, and introduce more subtle defects. When an entire team is generating more code simultaneously, the review bottleneck compounds. Individual throughput increases; system throughput stalls.

DORA 2025 identifies Value Stream Management as the practice that separates organizations capturing AI’s gains from those that don’t. VSM means mapping the full delivery workflow (code → review → test → deploy → customer value) and finding where local optimization is costing global throughput. A secondary finding from the same report: AI amplifies existing organizational patterns. Teams with strong delivery practices see capability multiply; teams with process gaps see technical debt accelerate. You can’t answer the ROI question without also answering the organizational health question.

What to measure #

Activity: qualify, don’t eliminate #

Activity metrics earn their keep as signals of engagement and friction: adoption depth, inner-loop versus outer-loop time, friction introduced by the toolchain itself. The key additions for AI-era measurement are two metrics that don’t appear in pre-AI dashboards:

AI Rework Ratio: the percentage of AI-generated code that requires significant edits within 30 days. Developers consistently report that AI outputs produce subtle, hard-to-detect errors that look sound but contain logic bugs. That makes rework a near-universal hidden cost, and one raw PR counts don’t surface.

Longitudinal AI Incident Rate: production incidents tied to AI-generated code at 30, 60, and 90 days post-merge. AI-generated code shows a 23.5% higher incident rate per PR compared to human-authored code, invisible to teams measuring only metadata.

Productivity: instrument quality alongside quantity #

The DORA five metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Mean Time to Recovery, and Rework Rate) remain the most validated framework for team-level delivery. In the AI context, they need to be read alongside quality-adjusted signals.

The DX Core 4’s concept of oppositional metrics is useful here: push Speed up without watching Quality and you get exactly the perverse incentives you’d expect, and a system that tracks only one side will get gamed on the other.

Impact: separate execution from market outcome #

Feature adoption metrics are the right bridge between developer execution and business outcome: breadth (percentage of eligible users adopting), depth (intensity of engagement), and duration (sustained use over 30/60/90 days). A feature achieving 70% adoption breadth represents demonstrable value, independent of whether the quarterly revenue number moved. A feature with 5% adoption and high churn might be an execution issue, or it might be a strategy one; the measurement system should make that distinction available rather than flatten it.

At the organizational level, useful AI ROI metrics include revenue per engineer, R&D as a percentage of revenue, and the percentage of engineering time on new capabilities versus maintenance. DX Core 4 research shows organizations using the framework achieve 14% increases in R&D time spent on feature development and 3–12% improvements in engineering efficiency, a concrete indicator of whether AI is freeing capacity to build. The connecting principle is correlation rather than attribution: linking AI adoption levels to organizational outcomes without holding individual developers accountable for macro outcomes outside their control.

Getting the instrumentation right #

Start with baselines, since measuring AI impact retroactively is working without a control. Capture pre-AI snapshots on DORA metrics, PR quality signals, and developer experience surveys at or before the start of AI adoption.

Separate AI-generated code from human-authored code in telemetry. Platforms measuring only PR metadata and commit volume can’t isolate AI impact from baseline changes; code-level instrumentation that tags AI-generated lines is the minimum for any credible ROI claim.

Finally, build a time-horizon-aware reporting cadence: activity signals in weekly dashboards, productivity signals in monthly retrospectives, and impact signals in quarterly reviews. Mixing these time horizons, running impact metrics weekly or presenting activity data as evidence of business outcomes, creates the measurement confusion the system is supposed to resolve.

Where attribution ends #

Individual developer metrics should connect to code quality, delivery reliability, and feature adoption. Team metrics should connect to DORA delivery performance and organizational throughput. Business outcomes (revenue, NPS, retention) belong at the organizational level, analyzed for correlation with AI adoption patterns rather than traced back to individuals.

The reason is causal rather than political: individual developers influence execution quality, execution quality influences feature adoption, and feature adoption influences retention and revenue. Each link is real and measurable, but only at the level of abstraction where the mechanism actually operates. Compressing the chain pushes accountability to the wrong level and makes the actual signals harder to read.

DORA, SPACE, and DX Core 4 have all converged on the same design principle: hold multiple dimensions simultaneously, instrument the tensions between them, and resist collapsing the chain into a single number, which is exactly what getting AI measurement right requires.