The Machine That Knew It Was a Metaphor

Weizenbaum wrote this book because he was frightened by his own creation. ELIZA, his simple pattern-matching chatbot from 1966, had convinced people it understood them — his secretary asked him to leave the room so she could speak privately with it — and the experience curdled something in him. By 1976 he had produced not a technical recantation but a philosophical one, arguing that computers are powerful metaphors for human reasoning but must never be mistaken for the thing itself. The distinction sounds quaint until you watch a therapist in 2025 recommend a CBT chatbot to a patient who can't afford weekly sessions, or until you read the fine print on an AI companion app marketed to lonely teenagers. Weizenbaum saw the substitution coming. He saw that the danger was not in the machine's capability but in our willingness to accept the performance as sufficient. What he could not have predicted — what would have staggered him — is that fifty years later the performance would be so much better and the willingness so much deeper, and that the two would feed each other in a loop no one seems interested in interrupting.

His central claim was that there exist decisions computers *ought not* make, regardless of whether they technically *can*. He was not talking about capability limits. He was talking about moral categories. A judge should not delegate sentencing to a machine, he argued, because judgment requires the judge to have lived as a human being, to have suffered, to carry the weight of consequence in a body. This was a hard sell in 1976, when the relevant computers filled rooms and ran on punch cards. It is a harder sell now, and for the opposite reason: the machines have become so fluent that the argument sounds like sentimentality. But predictive sentencing algorithms are deployed in American courtrooms. Autonomous weapons systems select targets. Insurance claim denials are automated at scale. Weizenbaum's line in the sand has not been erased so much as the sand itself has been paved over. His error, if it can be called that, was assuming the argument would be settled philosophically. It was settled economically, and the economy did not care about his categories.

What dates the book most visibly is its faith in a certain kind of institutional gatekeeping — the idea that scientists and engineers, if properly chastened, would decline to build what should not be built. Weizenbaum believed in professional conscience as a regulatory mechanism. He wrote as a tenured professor at MIT addressing other tenured professors, and his model of technological development assumed that the people making the decisions would be reachable by moral suasion. He could not have imagined a world in which the most consequential AI systems are built by private companies racing each other for market dominance, funded by venture capital that treats ethical hesitation as a competitive disadvantage. The absence of any serious engagement with capitalism as a force in technological development is the book's most conspicuous blind spot. He treats the computer as an idea when it was already becoming a commodity.

Still, certain passages land now with the force of delayed detonation. His insistence that a computer's "understanding" is always parasitic on human understanding — that it manipulates symbols without access to meaning — reads like a direct challenge to the large language model era, where the question of whether statistical pattern completion constitutes understanding has become the defining philosophical debate of the field. Weizenbaum did not anticipate transformers or attention mechanisms, but he anticipated the argument about them with uncanny precision. He sits in a lineage that runs from Norbert Wiener's *The Human Use of Human Beings* through to the work of Sherry Turkle, who studied under him, and forward to the AI ethics discourse that now employs thousands of people who have largely not read him. He gave the field its conscience, and the field put that conscience in a drawer.

Weizenbaum asked whether computers should be allowed to make certain human decisions. Fifty years later, the question the book raises is different and worse: now that they already do, and now that entire systems of labor, law, and care have been restructured around the assumption that they will continue to, what would it actually cost to take those decisions back?