In this episode of Future Proof, we sit down with Aman Khan, the Head of Product at Arize AI. Aman reveals why traditional product metrics fail for AI systems and shares Arize's framework for building evaluation systems that actually predict real-world AI performance, plus the emerging PM skills that separate successful AI products from failed experiments.

We discuss:

• How AI builders should think about evaluations
• The role of the AI pm and how product management is evolving
• How you should build with the expectations of foundation models changing.

(0:00) Highlights

(0:37) Intro

(1:40) What is an AI pm

(4:10) How PMs are evolving with AI

(8:10) The Aha moment in AI

(11:50) What AI builders should think about evaluations

(19:40) How AI builders best leverage their time in AI evaluations

(23:40) Prompt iteration - if your evaluations are not ideal, how do you iterate?

(27:40) What’s the minimum viable eval someone should write

(30:40) How would prioritization change based on the future of AI models

(36:40) Final thoughts

(38:00) Ethan's reflection

Ship integrations 7x faster https://www.useparagon.com/

Watch all Future Proof episodes: https://www.useparagon.com/future-proof