mind-crimes

Ethical concerns arising from the potential suffering of emulated minds or subagents within institutional control systems.

1 chapter across 1 book

Superintelligence: Paths, Dangers, Strategies (2014)Nick Bostrom

Chapter 11) but looks affordable in the context of a project that is not facing strong immediate competition. There would also be a cost in terms of the development time

This chapter discusses the challenges and potential costs associated with designing institutions composed of intelligent subagents, including emulations and artificial intelligences, to control superintelligent systems. It explores the ethical concerns such as mind crimes, the unpredictability of social structures among artificial agents, and the complexity added by institution design. The chapter concludes with a summary of various value-loading techniques, highlighting their strengths and weaknesses, and emphasizes the unresolved philosophical problem of determining which values to instill in a superintelligence.