The explicit goal of OpenAI, DeepMind and others is to create AGI.This is insanely risky.It keeps me up at night.AIs smarter than us might:🚨Resist shutdown.🚨Resist us changing their goals.🚨Ruthlessly pursue goals, even if they know it’s not what we want or intended.Some people think I’m nuts for believing this. But they often come round once they hear the central arguments.At the core of the AI doom argument are two big ideas:💡Instrumental Convergence💡The Orthogonality Thesis❌If you don’t understand these ideas, you won’t truly understand why some AI researchers are so worried about AGI or Superintelligence.Oxford philosopher Rhys Southan joined me to explain the situation.💡Rhys Southan and his co-authors Helena Ward and Jen Semler argue that powerful AIs might NOT resist having their goals changed. Possibly a fatal flaw in the Instrumental Convergence Thesis.This would be a BIG DEAL. It would mean we could modify powerful AIs if they go wrong.While I don’t fully agree with their argument, it radically changed how I understand the Instrumental Convergence Thesis and forced me to rethink what it means for AIs to have goals.Check out the paper "A Timing Problem for Instrumental Convergence" here: https://link.springer.com/article/10.1007/s11098-025-02370-4