Forget Superintelligence: Error-Prone AI Nearly Triggered a US-China War

Summary

Timnit Gebru and Emily M Bender argue that the immediate danger from AI is less likely to be a superintelligent system destroying humanity than people trusting unreliable systems in high-stakes settings. They focus on a CNN report published on 18 September, which has not been independently verified by other major outlets, describing a US military intelligence report that incorrectly claimed a Chinese ship was carrying nuclear-weapons components. American personnel reportedly prepared aircraft and soldiers to intercept the vessel, but officials discovered moments before execution that the report had been generated with a chatbot and contained false information. The authors say such an encounter could have escalated into a conflict between two nuclear powers. They describe today’s large language models as systems that generate plausible-looking text from training patterns, with well-documented failure modes rather than mysterious or superhuman agency. In their view, marketing and media narratives about superintelligence can encourage misplaced confidence in these tools, while concrete harms already arise when people treat them as authoritative. They cite errors involving hospital AI scribes and military intelligence systems as examples of foreseeable consequences. The authors call for regulation of AI use in military, medical and other life-or-death contexts, including evaluation within the intended use case, clear operating norms and accountability by empowered human decision-makers. They also argue that existing laws can be enforced against unsafe AI practices and urge journalists and lawmakers to scrutinize company claims instead of repeating them.