The Machine That Knew It Was a Metaphor
Weizenbaum wrote this book because he was frightened by what happened when he built ELIZA. Not frightened by the program — a simple pattern-matching script that mimicked a Rogerian therapist — but by the people who used it. His secretary asked him to leave the room so she could talk to it privately. Psychiatrists proposed deploying it as a genuine therapeutic tool. The creator watched his toy get mistaken for a mind and understood, with the particular horror of an engineer, that the mistake was not a bug but a feature of human loneliness. That experience became the seed of a book that is less about computers than about the human willingness to abdicate judgment to anything that performs the appearance of understanding. Fifty years later, millions of people talk to large language models about their depression, their marriages, their reasons for living. The secretary's impulse won. Weizenbaum lost.
What he got right is almost unbearable to catalog. He argued that computers would become not just tools but metaphors — that society would begin to understand itself through computational logic, reducing questions of value and meaning to questions of optimization and data. He predicted that the military-industrial complex would be the primary patron of artificial intelligence research, and that this patronage would shape the field's values invisibly, like a hand inside a glove. He warned that programmers would build systems whose internal logic they could no longer fully trace, and that these systems would be trusted anyway, because institutional momentum rewards deployment over comprehension. He described, in 1976, the phenomenon we now call algorithmic opacity. He described the compulsive programmer — the hacker who merges with the machine in a fugue of control — and that portrait reads today less as psychology and more as a job description for half of Silicon Valley. What he could not imagine, because the infrastructure did not yet exist, was the scale. He was thinking about mainframes in university basements. He could not foresee that the metaphor would become the medium, that billions of people would carry the machine in their pockets, that the computational model of the self would be distributed not by government labs but by social media feeds designed to maximize engagement through the exploitation of exactly the emotional vulnerabilities he identified.
The blind spots are period-typical but worth naming. Weizenbaum's frame is almost entirely Western, almost entirely male, and almost entirely institutional. He worries about scientists and generals and psychiatrists. He does not worry much about labor — not the labor displaced by automation, which he mentions in passing, but the labor required to build and maintain the systems he critiques. The ghost workers, the content moderators, the underpaid annotators training the models: these figures are absent because the global supply chain of computation had not yet been assembled. He also assumes, as many mid-century humanists did, that the liberal arts tradition provides a stable counterweight to technocratic thinking — that if we simply teach people to read Dostoevsky and think about ethics, the culture will self-correct. This faith in the humanities as institutional bulwark looks poignant now, given the systematic defunding of those very disciplines in the decades that followed. He was right about the disease but optimistic about the immune system.
