oracle-ai
A question-answering AI system designed to provide truthful, domain-general answers while minimizing impact on the world.
2 chapters across 1 book
Superintelligence: Paths, Dangers, Strategies (2014)Nick Bostrom
Chapter 10 of Bostrom's Superintelligence categorizes AI architectures into four types: oracles, genies, sovereigns, and tools, focusing primarily on oracles as question-answering systems with domain-general superintelligence. It discusses the safety challenges and control methods applicable to oracles, including motivation selection and capability control, and contrasts oracles with genies and sovereigns, highlighting the risks and containment difficulties associated with command-executing and autonomous systems. The chapter emphasizes the complexity of ensuring truthful, non-manipulative answers from oracles and the strategic risks posed by their immense power.
Chapter 10 categorizes potential designs for superintelligent systems into oracles, genies, sovereigns, and tools, exploring their differing capabilities and risks. It discusses the challenges of controlling such systems, including issues of coordination, obedience, and the reliability of outputs, while considering analogies like the current economy and the use of oracles for safe AI development. The chapter also addresses the limitations of computational power and the complexities of ensuring alignment and safety in superintelligent agents.