Whenever there was AI, there were benchmarks- from the turing test, to society-changing benchmarks like MNIST and ImageNet to modern problems like the ARC prize, benchmarked served a vital purpose to measure the performance of AI models. But something has shifted in modern times, in the LLM era have benchmarks lost their utility, becoming mere advertisement for big tech?
Even seemingly more sophisticated benchmarks like LM Arena can be gamed by tech giants. We also deep dive into healthcare benchmarks like OpenAI's Healthbench (deeply problematic) and Microsoft's AI-DXO orchestrator agent for diagnosis. Where is this all going? How do we make the perfect benchmark? Or is the real work to be done afterwards in the real world?
👋 Hey! If you are enjoying our conversations, reach out, share your thoughts and journey with us. Don't forget to subscribe whilst you're here :)
---
Timestamps
00:00 Intro - The OG benchmarks - Turing test, MNIST, ImageNET
06:40 Are large language models benchmarks similar to humans taking tests?
10:05 Are we testing model capability vs production ready?
12:00 LLM era - data contamination
15:30 LM Arena - The leaderboard illusion paper - how big tech games benchmarks
28:35 Goodhart's law - When a measure becomes a target, it ceases to be a good measure
32:05 Some good benchmarks - games - Pokemon, ARC prize, Minecraft
34:35 Medical benchmarks - OpenAI's healthbench has some big problems
46:50 Microsoft AI-DXO orchestrator for case reports
---
Connect with Us
Your Hosts:
👨🏻⚕️ Doc - Dr. Joshua Au Yeung - LinkedIn
🤖 Dev - Zeljko Kraljevic - Twitter
Follow & Subscribe:
YT: https://youtube.com/@DevAndDoc
Spotify: Follow us on Spotify
Apple Podcasts: Listen on Apple Podcasts
Substack: https://aiforhealthcare.substack.com/
For enquiries:
📧 Devanddoc@gmail.com
---
Production Credits
🎞️ Editor: Dragan Kraljević - Instagram
🎨 Brand & Art: Ana Grigorovici - Behance
