coherent-extrapolated-volition
Yudkowsky's proposal that a superintelligence should act according to humanity's idealized and collectively extrapolated desires, reflecting what we would want if we were more informed, rational, and unified.
2 chapters across 1 book
Superintelligence: Paths, Dangers, Strategies (2014)Nick Bostrom
Chapter 13 addresses the challenge of selecting the final value or goal to install in a superintelligence, emphasizing the difficulty and risks of making such a choice based on current human moral understanding. It introduces the concept of indirect normativity as a strategy to delegate the complex task of value selection to the superintelligence itself, anchored by abstract conditions rather than fixed, potentially flawed human values. The chapter discusses Eliezer Yudkowsky's coherent extrapolated volition (CEV) as a prototype for indirect normativity, explaining how it aims to approximate humanity's idealized collective wishes through a process of extrapolation and consensus.
Chapter 13 explores the philosophical and ethical foundations for selecting criteria to guide superintelligent AI decision-making, focusing on normative ethics, metaethics, and moral epistemology. It discusses various ethical frameworks such as consequentialism, deontology, and virtue ethics, and addresses challenges like moral uncertainty, the role of coherent extrapolated volition (CEV), and the potential for AI to interpret and act upon complex moral judgments. The chapter also considers practical issues such as deferring to AI moral judgment, the possibility of moral realism, and the implications of building AI systems that act according to moral principles.