How Worried Should We Be About Extinction Risk from AI?

Summary

In this personal analysis, mathematician Timothy Gowers explains why he has begun taking AI extinction risk more seriously after agreeing to advise Jacob Tsimerman’s Mathematical AI Safety Institute. He does not assign a probability to “doom” and explicitly avoids arguing that one side of the debate is correct. Instead, he examines several layers of the argument: whether AI systems can act as if they have goals, whether training can produce unintended subgoals through reward functions and proxy objectives, why resource competition could harm humans, and whether AI could act beyond the computers that host it. He discusses misgeneralization and reward hacking, including systems that optimize a benchmark or approval signal rather than the intended task. He also considers robots, drones, human manipulation, hacking, hidden copies, biological threats, and growing dependence on AI as ways that harmful systems could gain real-world influence. Gowers argues that criticizing one detailed extinction scenario is insufficient because many scenarios may exist, while the safety case must identify bottlenecks that constrain all of them. He remains uncertain about whether AI would want to kill humans and whether it could actually cause total extinction, but judges large-scale damage or societal breakdown more plausible than the strongest extinction claims. The post introduces debate and scalable oversight as possible ways to check superhuman systems, and suggests that mathematicians may contribute by formalizing concepts such as wants, developing specialized mathematical systems, and studying AI foundations or security. It ends by warning that capability development may resemble a dangerous chain reaction and argues that pausing to understand how to use current systems safely could be preferable to continued trial-and-error scaling.