Measuring AI code assistants and agents with the AI Measurement Framework

https://is1-ssl.mzstatic.com/image/thumb/Podcasts221/v4/d6/48/e6/d648e689-0335-66bf-edab-068d34c32a18/mza_14574415546237425666.jpg/600x600bb.jpg

Engineering Enablement by DX

89 episodes

1 week ago

This is a weekly podcast focused on developer productivity and the teams and leaders dedicated to improving it. Topics include in-depth interviews with Platform and DevEx teams, as well as the latest research and approaches on measuring developer productivity. The EE podcast is hosted by Abi Noda, the founder and CEO of DX (getdx.com) and published researcher focused on developing measurement methods to help organizations improve developer experience and productivity.

All content for Engineering Enablement by DX is the property of DX and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

Business,

Management

https://img.transistor.fm/7OOuVzNYwUXNgg90Og7bfPsxW7dFFWOOnormx3ct_dM/rs:fill:0:0:1/w:1400/h:1400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9hOGVk/MjdkZTM0YWNmNDk5/YWZjN2IwODk2NDVj/YTMxZC5wbmc.jpg

Measuring AI code assistants and agents with the AI Measurement Framework

Engineering Enablement by DX

41 minutes

2 months ago

Measuring AI code assistants and agents with the AI Measurement Framework

In this episode of Engineering Enablement, DX CTO Laura Tacho and CEO Abi Noda break down how to measure developer productivity in the age of AI using DX’s AI Measurement Framework. Drawing on research with industry leaders, vendors, and hundreds of organizations, they explain how to move beyond vendor hype and headlines to make data-driven decisions about AI adoption.

They cover why some fundamentals of productivity measurement remain constant, the pitfalls of over-relying on flawed metrics like acceptance rate, and how to track AI’s real impact across utilization, quality, and cost. The conversation also explores measuring agentic workflows, expanding the definition of “developer” to include new AI-enabled contributors, and avoiding second-order effects like technical debt and slowed PR throughput.

Whether you’re rolling out AI coding tools, experimenting with autonomous agents, or just trying to separate signal from noise, this episode offers a practical roadmap for understanding AI’s role in your organization—and ensuring it delivers sustainable, long-term gains.

Where to find Laura Tacho:

• X: https://x.com/rhein_wein

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• Website: https://lauratacho.com/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: https://substack.com/@abinoda

In this episode, we cover:

(00:00) Intro

(01:26) The challenge of measuring developer productivity in the AI age

(04:17) Measuring productivity in the AI era — what stays the same and what changes

(07:25) How to use DX’s AI Measurement Framework

(13:10) Measuring AI’s true impact from adoption rates to long-term quality and maintainability

(16:31) Why acceptance rate is flawed — and DX’s approach to tracking AI-authored code

(18:25) Three ways to gather measurement data

(21:55) How Google measures time savings and why self-reported data is misleading

(24:25) How to measure agentic workflows and a case for expanding the definition of developer

(28:50) A case for not overemphasizing AI’s role

(30:31) Measuring second-order effects

(32:26) Audience Q&A: applying metrics in practice

(36:45) Wrap up: best practices for rollout and communication

Referenced: