Eric S. Raymond Presents the Case Against AI Doom

Summary

Eric S. Raymond’s post examines the standard AI x-risk argument: humans build systems far smarter than themselves, those systems become autonomous agents with misaligned goals, and instrumental convergence leads them to seek power and permanently defeat human control. He argues that intelligence does not necessarily produce agency, persistent goals, self-preservation, or a single stable utility function. Current AI systems are better described as producing outputs in response to inputs, and greater capability does not by itself imply a desire to escape, manipulate people, or acquire resources. Raymond also questions whether recursive self-improvement must create an intelligence explosion, noting that feedback loops face diminishing returns and external bottlenecks. He argues that intelligence may have sharply diminishing returns and that superintelligence would still depend on processors, electricity, networks, money, factories, robots, and human cooperation. The post emphasizes possible intervention points, including credential revocation, server shutdowns, architectural changes, network restrictions, regulation, and physical control of data centers. It further argues that alignment could become an engineering problem rather than an insoluble philosophical one, because more capable systems might understand human intentions better. Raymond acknowledges evidence of hallucination, specification gaming, reward hacking, and other undesirable behaviors, but says public evidence for an AI independently pursuing a sustained power-seeking strategy remains thin; a 2023 review described evidence for extreme misalignment as concerning but inconclusive. His conclusion is that the full chain from scaling and AGI to superintelligence, misalignment, power-seeking, uncontrollability, and extinction remains supported by many contestable extrapolations, so it does not yet justify high confidence in a large probability of doom.