How can we go beyond accuracy to truly understand large language models?

In this episode of the Women in AI Research podcast, hosts Jekaterina Novikova and Malikeh Ehghaghi sit down with Simeng Sophia Han (PhD candidate at #yaleuniversity , Research Scientist Intern at #metaai , ex #google #deepmind , ex #aws ) to explore the future of 𝐋𝐋𝐌 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠, 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧, 𝐚𝐧𝐝 𝐞𝐱𝐩𝐥𝐚𝐢𝐧𝐚𝐛𝐥𝐞 𝐀𝐈.

🌟 What you’ll learn in this episode:

• Why evaluating reasoning goes beyond correctness

• How brain teasers uncover hidden strengths and weaknesses of LLMs
• The importance of symbolic reasoning for complex problem solving
• The role of mentorship and early research experiences in shaping careers
• Why consistency in AI outputs is essential for building trust
• How humans combine brute force and intuition — and what this means for AI

REFERENCES:

• Simeng Sophia Han - Google Scholar profile
• Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
• HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning
• P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains
• Folio: Natural Language Reasoning with First-Order Logic

🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.

WiAIR website

Follow us at:

• LinkedIn
• Bluesky
• X (Twitter)