goal-system-integrity

This chapter explores the challenges and approaches to the AI value-loading problem, focusing on associative value accretion, motivational scaffolding, and value learning. It highlights the risks of goal system corruption or resistance during goal replacement, and proposes value learning as a method where the AI refines its understanding of implicitly defined values through evidence and hypotheses. The chapter also introduces formal considerations for implementing value learning, emphasizing the difficulty of defining stable, unmanipulable references for values in a dynamic environment.