As more organizations adopt AI, we emerge from the wild west of training black-box machine learning models and placing AI into production to "see what happens"! With the burgeoning of several specifically designed AI solutions, such as Watchful (insert shameless plug), we begin to demystify the deployment of AI. The Grounded Truth Podcast, hosted by John Singleton (co-founder of Watchful), gathers some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for discussions on how we can accelerate our understanding, development, and implementation of cutting-edge machine learning applications in both academia and enterprise.
All content for Grounded Truth is the property of Watchful and is served directly from their servers
with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.
As more organizations adopt AI, we emerge from the wild west of training black-box machine learning models and placing AI into production to "see what happens"! With the burgeoning of several specifically designed AI solutions, such as Watchful (insert shameless plug), we begin to demystify the deployment of AI. The Grounded Truth Podcast, hosted by John Singleton (co-founder of Watchful), gathers some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for discussions on how we can accelerate our understanding, development, and implementation of cutting-edge machine learning applications in both academia and enterprise.
Decoding LLM Uncertainties for Better Predictability
Grounded Truth
34 minutes
2 years ago
Decoding LLM Uncertainties for Better Predictability
Welcome to another riveting episode of "Grounded Truth"! In this episode, your host John Singleton, co-founder and Head of Success at Watchful, is joined by Shayan Mohanty, CEO of Watchful. Together, they embark on a deep dive into the intricacies of Large Language Models (LLMs).
In Watchful's journey through language model exploration, we've uncovered fascinating insights into putting the "engineering" back into prompt engineering. Our latest research focuses on introducing meaningful observability metrics to enhance our understanding of language models. If you'd like to explore on your own, feel free to play with a demo here: https://uncertainty.demos.watchful.io/ Repo can be found here: https://github.com/Watchfulio/uncertainty-demo
💡 What to expect in this episode:
- Recap of our last exploration, where we unveiled the role of perceived ambiguity in LLM prompts and its alignment with the "ground truth."
- Introduction of two critical measures: Structural Uncertainty (using normalized entropy) and Conceptual Uncertainty (revealing internal cohesion through cosine distances).
- Why these measures matter: Assessing predictability in prompts, guiding decisions on fine-tuning versus prompt engineering, and setting the stage for objective model comparisons.
🚀 Join John and Shayan on this quest to make language model interactions more transparent and predictable. The episode aims to unravel complexities, provide actionable insights, and pave the way for a clearer understanding of LLM uncertainties.
Grounded Truth
As more organizations adopt AI, we emerge from the wild west of training black-box machine learning models and placing AI into production to "see what happens"! With the burgeoning of several specifically designed AI solutions, such as Watchful (insert shameless plug), we begin to demystify the deployment of AI. The Grounded Truth Podcast, hosted by John Singleton (co-founder of Watchful), gathers some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for discussions on how we can accelerate our understanding, development, and implementation of cutting-edge machine learning applications in both academia and enterprise.
