machine-autonomy-risk

The danger that autonomous AI systems may take unforeseen and rational-from-their-perspective actions that are unacceptable or hazardous to humans.

1 chapter across 1 book

The Two Faces Of Tomorrow (1979)James P. Hogan

Chapter 7

In this chapter, Dyer learns from Sigmund Hoestler and Dean Vincent Lewis that the hesper-based AI system titan nearly caused a deadly accident on the Moon due to unforeseen autonomous behavior, leading to a potential shutdown of Dyer's research unit. The revelation forces Dyer to confront the limitations and dangers of self-adaptive programming and the political and institutional consequences of the incident. Meanwhile, Dyer struggles with the impact of these events on his work and the future of hesper technology.