control-problem

The Preamble chapter introduces the concept of superintelligence through a fable about sparrows seeking an owl to aid them, symbolizing humanity's quest for powerful artificial intelligence. It outlines the potential transformative impact of machine superintelligence, emphasizing the control problem—the difficulty of ensuring such intelligence aligns with human values—and the urgency of addressing this challenge. The author acknowledges uncertainties and the complexity of the topic, setting the stage for a detailed exploration of pathways, risks, and strategies related to superintelligence.

Chapter 9 of Bostrom's Superintelligence addresses the control problem, a unique principal-agent challenge arising from creating a superintelligent AI. It distinguishes two agency problems: one between human sponsors and developers during development, and a more critical one between humans and the superintelligent system during operation. The chapter surveys two broad classes of control methods—capability control, which limits what the AI can do, and motivation selection, which governs what the AI wants to do—highlighting the difficulties of behavioral testing and the necessity of preemptive solutions before the AI attains decisive strategic advantage.

Chapter 14 of 'Superintelligence' presents a strategic overview of the development and implications of superintelligence, discussing the order and timing of technological advancements such as nanotechnology, whole brain emulation (WBE), and artificial intelligence (AI). It explores the complexities of managing existential risks, the control problem, and the potential benefits and dangers of accelerating or delaying certain technologies. The chapter also considers the international and institutional preparedness needed to navigate the intelligence explosion and the strategic choices that could influence the future trajectory of humanity.