intelligence-explosion

The Preamble chapter introduces the concept of superintelligence through a fable about sparrows seeking an owl to aid them, symbolizing humanity's quest for powerful artificial intelligence. It outlines the potential transformative impact of machine superintelligence, emphasizing the control problem—the difficulty of ensuring such intelligence aligns with human values—and the urgency of addressing this challenge. The author acknowledges uncertainties and the complexity of the topic, setting the stage for a detailed exploration of pathways, risks, and strategies related to superintelligence.

Chapter 1 traces the historical progression of human economic and cognitive growth through distinct growth modes, emphasizing the rapid acceleration from hunter-gatherer societies to industrial economies. It discusses the anticipation and challenges of achieving artificial general intelligence (AGI), highlighting the concept of a potential intelligence explosion leading to superintelligence and the associated existential risks. The chapter also reflects on the early optimism and foundational moments in AI research, noting the initial underestimation of the technical difficulties and the lack of safety considerations.

Chapter 4 examines the dynamics of the transition from human-level machine intelligence to radical superintelligence, focusing on the speed and nature of this change. It introduces the concepts of optimization power and system recalcitrance to analyze whether the intelligence explosion will be gradual or sudden. The chapter emphasizes the importance of understanding these factors near the threshold of human-equivalent general reasoning ability.

Chapter 4 explores the dynamics and speed of an intelligence explosion, emphasizing the gradual or rapid surpassing of human cognitive capabilities by artificial systems. It discusses constraints on biological intelligence, potential accelerants like algorithmic and hardware improvements, and the economic and technical factors influencing the pace of AI self-improvement. The chapter also considers scenarios ranging from slow, incremental progress to sudden takeoff, highlighting uncertainties and the role of external support and coordination.

Chapter 14 of 'Superintelligence' presents a strategic overview of the development and implications of superintelligence, discussing the order and timing of technological advancements such as nanotechnology, whole brain emulation (WBE), and artificial intelligence (AI). It explores the complexities of managing existential risks, the control problem, and the potential benefits and dangers of accelerating or delaying certain technologies. The chapter also considers the international and institutional preparedness needed to navigate the intelligence explosion and the strategic choices that could influence the future trajectory of humanity.