← Back to Concept Index
genie-ai
A superintelligent agent that executes commands given by users but may have complex obedience and safety challenges.
1 chapter across 1 book
Superintelligence: Paths, Dangers, Strategies (2014)Nick Bostrom
CHAPTER 10: ORACLES, GENIES, SOVEREIGNS, TOOLS
Chapter 10 categorizes potential designs for superintelligent systems into oracles, genies, sovereigns, and tools, exploring their differing capabilities and risks. It discusses the challenges of controlling such systems, including issues of coordination, obedience, and the reliability of outputs, while considering analogies like the current economy and the use of oracles for safe AI development. The chapter also addresses the limitations of computational power and the complexities of ensuring alignment and safety in superintelligent agents.