How to Measure AI Productivity Gains and ROI
James Shore argues that engineering leaders should assess AI by comparing delivery benefits with token, labor, maintenance, and change-management costs rather than relying on anecdotes or code-volume metrics. He cites a CIO at a 170-person organization who reported a 2.1x productivity increase; top AI users at that organization reportedly spent $21,000 per month on tokens, while another engineering leader reportedly spent $6,000–$8,000 per engineer each month. Shore presents these figures through anecdotal accounts and recommends making cost assumptions explicit, modeling optimistic and pessimistic scenarios, and measuring AI approaches rather than individual developers.
Shore’s framework gives engineering leaders a practical way to compare AI delivery gains with token, personnel, transition, and maintenance costs, while emphasizing that anecdotes should be supplemented by explicit measurement and business-outcome modeling.
This source provides an excerpt. Continue reading on the original site.