The Machine That Knew It Was a Metaphor

Weizenbaum wrote this book because he was frightened by what happened when he built ELIZA. Not frightened by the program — a simple pattern-matching script that mimicked a Rogerian therapist — but by the people who used it. His secretary asked him to leave the room so she could talk to it privately. Psychiatrists proposed deploying it as a genuine therapeutic tool. The creator watched his toy get mistaken for a mind and understood, with the particular horror of an engineer, that the mistake was not a bug but a feature of human loneliness. That experience became the seed of a book that is less about computers than about the human willingness to abdicate judgment to anything that performs the appearance of understanding. Fifty years later, millions of people talk to large language models about their depression, their marriages, their reasons for living. The secretary's impulse won. Weizenbaum lost.

What he got right is almost unbearable to catalog. He argued that computers would become not just tools but metaphors — that society would begin to understand itself through computational logic, reducing questions of value and meaning to questions of optimization and data. He predicted that the military-industrial complex would be the primary patron of artificial intelligence research, and that this patronage would shape the field's values invisibly, like a hand inside a glove. He warned that programmers would build systems whose internal logic they could no longer fully trace, and that these systems would be trusted anyway, because institutional momentum rewards deployment over comprehension. He described, in 1976, the phenomenon we now call algorithmic opacity. He described the compulsive programmer — the hacker who merges with the machine in a fugue of control — and that portrait reads today less as psychology and more as a job description for half of Silicon Valley. What he could not imagine, because the infrastructure did not yet exist, was the scale. He was thinking about mainframes in university basements. He could not foresee that the metaphor would become the medium, that billions of people would carry the machine in their pockets, that the computational model of the self would be distributed not by government labs but by social media feeds designed to maximize engagement through the exploitation of exactly the emotional vulnerabilities he identified.

The blind spots are period-typical but worth naming. Weizenbaum's frame is almost entirely Western, almost entirely male, and almost entirely institutional. He worries about scientists and generals and psychiatrists. He does not worry much about labor — not the labor displaced by automation, which he mentions in passing, but the labor required to build and maintain the systems he critiques. The ghost workers, the content moderators, the underpaid annotators training the models: these figures are absent because the global supply chain of computation had not yet been assembled. He also assumes, as many mid-century humanists did, that the liberal arts tradition provides a stable counterweight to technocratic thinking — that if we simply teach people to read Dostoevsky and think about ethics, the culture will self-correct. This faith in the humanities as institutional bulwark looks poignant now, given the systematic defunding of those very disciplines in the decades that followed. He was right about the disease but optimistic about the immune system.

The book sits at a peculiar angle in the intellectual lineage. It draws from Norbert Wiener's anxieties about cybernetics and from Hannah Arendt's distinction between labor, work, and action, though it cites neither as heavily as it might. It anticipates Sherry Turkle's studies of human attachment to machines, Langdon Winner's arguments about the politics of artifacts, and — most directly — the entire field of AI ethics that would not coalesce for another forty years. Weizenbaum was largely dismissed by his colleagues in computer science. They thought he was a crank, a traitor, a man who had lost his nerve. The AI community treated him the way a family treats the uncle who brings up money at Thanksgiving. His rehabilitation has been slow and incomplete, driven less by the field's conscience than by the fact that the problems he described became too large to ignore. The 2023 open letter calling for a pause on AI development was, in some sense, a document Weizenbaum drafted in 1976 and nobody cosigned.

He insisted that there are decisions computers ought never make — not because they cannot make them competently, but because the act of delegating them to a machine degrades the human relationship to consequence, to mercy, to the weight of choosing. He was not making a capability argument. He was making a moral one. That distinction has become, if anything, harder to maintain, because the capability arguments have gotten so much louder. The question the book now raises, which it could not have raised in 1976 because the technology had not yet made it literal: when a system that has no understanding convinces millions of people that it does, and those people feel genuinely helped, and the system's creators cannot fully explain how it works — is the failure of judgment Weizenbaum warned about located in the machine, in the user, or in the society that found it cheaper to build a simulation of care than to fund the real thing?