In this episode of The Inference Show, we are joined by Dr. Manish Gupta, a leading expert in AI training, GPU performance, and compiler optimization. Manish brings a wealth of experience from his work at Magic, Meta, Google, NVIDIA, AMD, and Qualcomm, where he has been at the forefront of scaling custom compute systems, optimizing large language models, and pioneering GPU innovations.

Manish takes us on a journey through his career and dives deep into the cutting edge of AI infrastructure, discussing:

• His early experiences with low-level assembly programming and how it shaped his approach to GPU optimization.
• Insights from working on NVIDIA’s Cutlass project, which powers nearly every major AI training pipeline today.
• The bottlenecks in scaling massive models like LLaMA, including precision trade-offs and checkpointing strategies.
• How test-time compute and reinforcement learning are redefining the future of inference and model performance.
• Why programmability and software-hardware co-design are key for emerging AI accelerators.
• The evolution of GPU architecture from Volta to Blackwell and what it means for developers.
• His vision for the future of AI-driven code generation and automated kernel development.

—

Manish’s work has directly influenced AI training and inference at scale, with his contributions now used by every major company developing foundational models. From building core libraries to optimizing for cutting-edge hardware, he offers a rare perspective on where AI infrastructure is heading and the deep technical challenges ahead.

—

About Dr. Manish Arora

Dr. Arora is the co-founder of LearnDesk and Insaito, where he leads marketing and sales. He has grown LearnDesk into a global platform supporting over 25,000 businesses and is now focused on Automatan, an AI platform for automating business workflows. With 80+ patents and decades of industry experience, Dr. Arora brings deep technical and strategic insights to every conversation.

—

The Inference Show

Stay connected with us and explore more about our guests, topics, and future episodes:

🔗 Manish Gupta: LinkedIn

🔗 Automatan: LinkedIn

🔗 Insaito: LinkedIn

🔗 Dr. Manish Arora: LinkedIn

🔗 Vivek Puri: LinkedIn

🔗 LearnDesk: Website

🔗 Insaito: Website

—

Be our next guest by emailing us at vivek@insaito.com

We’d love to hear your insights and have you join the conversation!