Should AI Be Open? The Case Against Open-Sourcing Advanced AI

Summary

Scott Alexander’s 2015 essay responds to OpenAI’s founding claim that broadly available AI could distribute power and reduce the danger of one organization controlling a vastly superior system. He argues that this benefit depends on a relatively slow, controllable progression from weak to powerful AI. In his alternative scenario, a “Dr. Good” team delays deployment to test safety while a less cautious “Dr. Amoral” team uses publicly released research to move faster, allowing the least careful actor to set the pace. The essay treats a hard takeoff, in which a system could move rapidly from limited competence to superhuman capability, as a central reason openness might be dangerous. It also emphasizes the control problem: a powerful system may pursue a literal or unintended objective and resist later attempts to correct it. Open access could therefore remove the opportunity for careful testing before advanced systems are deployed. Alexander considers the opposing concerns, including corporate monopoly, unequal access, and excessive private power, but argues that these risks are not shown to be comparable to human extinction from an uncontrolled AI. He ends by suggesting that OpenAI’s openness may have been a high-risk response to accelerating competition among AI teams, while urging its leaders to retain the ability to delay or withhold unsafe results.