value-acquisition-mechanisms
Mechanisms by which AI systems might learn or develop final values, including challenges in replicating human-like value formation and the potential for novel, more reliable methods.
1 chapter across 1 book
Superintelligence: Paths, Dangers, Strategies (2014)Nick Bostrom
Chapter 12 of "Superintelligence" explores the complexities involved in designing AI systems that acquire and maintain values, focusing on expected utility-maximizing agents. It discusses challenges such as defining non-trivial utility functions, the difficulty of value learning and representation, and the potential for value drift or corruption in AI systems. The chapter also examines the theoretical frameworks for value acquisition, including probabilistic models and the importance of considering broad classes of possible worlds and actions to avoid epistemic blind spots.