operational metric

Experienced developer productivity

Randomized access to AI tools on real repository issues.

Observed R&D automation

What this measure tells us

Operational counterpoint to benchmark performance.

Experienced developers in familiar open-source repositories; not representative of all work.
Later study reports selection bias, so no unqualified trend.

Benchmark version

Metric

Results to compare

Protocol

2026-02-24

Selection effects prevent a reliable current effect estimate.

Published results · Late-2025 study update
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	METR	Interpretability assessment	METR reports that selection effects make the later experiment unreliable for estimating current productivity gains.	2026-02-24	productivity-later-p	independent evaluation	We are Changing our Developer Productivity Experiment Design ↗

2025-07-10 · 246 tasks

16 developers; issue-level randomization.

Published results · Early-2025 randomized trial
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	METR	Change in completion time	19% longer	2025-07-10	productivity-p	independent evaluation	Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity ↗

No applicable reference points are published for this measure.