James Shore Proposes a Rigorous Way to Measure AI’s Impact on Software Delivery Speed

Summary

In the second part of his series on quantifying AI’s impact on software development, James Shore focuses on delivery speed rather than productivity as a whole. He defines productivity as value produced divided by money spent, while asking the narrower question of whether teams finish assigned work faster. Shore argues that no objective cross-team speed metric exists because features, bugs, stories, estimates, and other outputs vary in size, and estimates are commonly over-optimistic. His proposed solution is to compare people or teams with themselves and run a randomized within-subject repeated-measures trial. Before workers know which method they will use, they should estimate each task, then be randomly assigned to a clearly defined “with AI” or “without AI” approach. The two approaches should include the same workflow improvements, so streamlined processes are not incorrectly credited to AI. Teams should decide how to attribute rework, whether to measure calendar time or effort, and how to handle parallel work, while collecting estimated time, actual time, task type, and spending; optional measures include bugs, incidents, mental energy, and excess code. Each participant’s results can then be compared across conditions and aggregated with statistical help. Shore notes that a reported 2x coding-speed gain becomes only a 1.33x overall improvement when coding represents 50% of a person’s week. He argues that measurement is justified by large differences in AI spending and cites a METR study in which engineers overestimated AI’s speed benefits by nearly 50%. The article concludes that delivery speed alone does not establish productivity or business value, and that the next installment will address unintended consequences.