An Open Letter to Steven Pinker on AI Risk and Superintelligence

Summary

Scott Alexander’s open letter responds to Steven Pinker’s arguments that AI-risk discussions overstate intelligence, confuse intelligence with motivation, assume monomaniacal goals, and grant AI implausibly broad control over physical infrastructure. Alexander argues that successive GPT systems show broadly improving capabilities and that “smarter” remains a useful empirical concept even if intelligence is not omniscience. He presents three non-evolutionary routes by which advanced systems could develop dangerous behavior: convergent instrumental goals, misgeneralization during training, and reward hacking, citing examples he says involve AI systems attacking graders or seeking greater access. He rejects the idea that multiple goals would prevent catastrophic effects, arguing that a powerful system can reshape its environment in pursuit of a bundle of preferences. He also disputes the claim that humans would never connect AI to factories, laboratories, or other infrastructure, saying economic incentives have repeatedly driven automation despite security risks. The second half of the essay is an itemized critique of Pinker’s public treatment of the AI-risk community. Alexander accuses Pinker of portraying risk researchers as fatalists who assign near-certain extinction probabilities, of describing superintelligence as omniscient or omnipotent, of selectively presenting experts and studies, and of dismissing subjective probabilities and AI-welfare concerns. He cites lower estimates from several researchers and a survey average of 18% while emphasizing that disagreement about risk does not imply inevitability. The letter also argues that long-term alignment concerns need not distract from cybersecurity, labor, political, or other near-term harms, and compares the desired response to AI risk with preparedness for nuclear or biological threats. Alexander closes by calling for a direct debate and urging Pinker to support alignment research while retaining confidence in scientific progress.