How do you know if your AI product is actually any good? Hamel Husain has been answering that question for over 25 years. As a former machine learning engineer and data scientist at Airbnb and GitHub (where he worked on research that paved the way for GitHub Copilot), Hamel has spent his career helping teams debug, measure, and systematically improve complex systems. In this episode, Hamel joins Teresa Torres to break down the craft of error analysis and evaluation for AI products. Together, they trace his journey from forecasting guest lifetime value at Airbnb to consulting with startups like Nurture Boss, an AI-native assistant for apartment complexes. Along the way, they dive into: - Why debugging AI starts with thinking like a scientist - How data leakage undermines models (and how to spot it) - Using synthetic data to stress-test failure modes - When to rely on code-based assertions vs. LLM-as-judge evals - Why your CI/CD set should always include broken cases - How to prioritize failure modes without drowning in them Whether you’re a product manager, engineer, or designer, this conversation offers practical, grounded strategies for making your AI features more reliable—and for staying sane while you do it.

How do you build an AI-powered assistant that teachers will actually use? In this episode of Just Now Possible, Teresa Torres talks with Thom van der Doef (Principal Product Designer), Mary Gurley (Director of Learning Design & Product Manager), and Ray Lyons (VP of Product & Engineering) from eSpark. Together, they’ve spent more than a decade building adaptive learning tools for K–5 classrooms—and recently launched an AI-powered Teacher Assistant that helps educators align eSpark’s supplemental lessons with district-mandated core curricula. We dig into the real story behind this feature: - How post-COVID shifts in education created new pressures for teachers and administrators - Why their first instinct—a chatbot interface—failed in testing, and what design finally worked - The technical challenges of building their first RAG system and learning to wrangle embeddings - How their background in education shaped a surprisingly rigorous eval process, long before “evals” became a buzzword - What they’ve learned from thousands of teachers using the product this school year It’s a detailed look at the messy, iterative process of building AI-powered products in the real world—straight from the team doing the work.

When ChatGPT launched, Stack Overflow faced a cataclysmic shift: developer behavior was changing overnight. In this episode, Teresa Torres talks with Ellen Brandenburger, former product leader at Stack Overflow, about how her team navigated the disruption, prototyped AI features, and eventually built an entirely new business line. Ellen shares the inside story of Overflow AI—from the first scrappy prototypes of conversational search, through multiple iterations with semantic search and RAG, to the tough decision to roll the product back when it couldn’t meet developer standards. She also explains how Stack Overflow turned a looming threat into opportunity by creating technical benchmarks and licensing its Q&A corpus to AI labs. This episode offers a rare look at what it really takes to adapt when a platform-defining shift hits—and what product managers, designers, and engineers can learn about prototyping, evaluating quality, and building in uncertainty.

How AI products come to life—straight from the builders themselves. In each episode, we dive deep into how teams spotted a customer problem, experimented with AI, prototyped solutions, and shipped real features. We dig into everything from workflows and agents to RAG and evaluation strategies, and explore how their products keep evolving. If you’re building with AI, these are the stories for you. The first full episode drops on Thursday, September 18th. Don't miss it!