Concrete Problems in AI Safety

Summary

This 2016 paper examines accidents in machine-learning systems, defined as unintended and harmful behavior caused by poor design in real-world AI systems. It organizes five practical safety research problems around three sources of risk: incorrect objectives, objectives that are too costly to evaluate often, and undesirable behavior during learning. The first group includes avoiding side effects and avoiding reward hacking. The second is scalable supervision, which addresses situations where evaluating an objective frequently is too expensive. The third includes safe exploration and distributional shift, covering risks that arise while systems learn or encounter conditions unlike those represented during development. The authors review earlier work in these areas and propose research directions intended to remain relevant to advanced AI systems. The paper concludes by considering how to reason productively about the safety of future AI applications.