Blog · Review Essay · Last reviewed June 25, 2026

God & Golem, Inc. and the Ethics of Machine Obedience

Norbert Wiener's God & Golem, Inc. is a short, late book by the founder of cybernetics, written where machine learning, self-reproducing machinery, automation, game-playing programs, and religious metaphor had begun to converge. Its usefulness now is not that it predicts every modern AI technique. It is that it names an older danger with unusual clarity: a machine may do what it was set up to do, and that obedience can become the form in which human responsibility disappears.

For this review, machine obedience means target-bound execution: a technical system carries a command, incentive, prompt, policy, or permission into the world without the judgment needed to know whether the command was worthy, complete, or still legitimate. It is not obedience in a moral or legal sense. It is operational obedience: authority converted into action through metrics, interfaces, data, prompts, permissions, APIs, feedback, and workflow defaults.

The operational test is whether obedience leaves enough resistance and memory: source of command, scope of authority, evidence used, action taken, affected party, human override, and appeal path. If those cannot be reconstructed, the system has not merely automated a task. It has made responsibility harder to find.

The Book

God & Golem, Inc.: A Comment on Certain Points where Cybernetics Impinges on Religion was published by the MIT Press in 1964, with a paperback edition in 1966. MIT Press lists both editions at 99 pages, records the paperback ISBN as 9780262730112, and notes that the book is based on lectures given at Yale, at the Societe Philosophique de Royaumont, and elsewhere. MIT Press Direct hosts an open-access edition, and the National Book Foundation records the book as the 1965 National Book Award winner for Science, Philosophy, and Religion.

Wiener was not writing from the edge of the field. MIT Press identifies him as an MIT mathematics faculty member from 1919 until his death in 1964, the coiner of "cybernetics," and a 1963 National Medal of Science recipient. Cybernetics had given a technical language to control and communication in animals and machines. The Human Use of Human Beings had turned that language toward public ethics. God & Golem, Inc. adds a sharper mythic frame: when humans build systems that learn, reproduce, and obey, they should stop pretending that command is morally simple.

The book is brief enough to seem slight, but its compression is part of its force. It asks three connected questions. What does it mean for a machine to learn? What does it mean for machines to make other machines in their own image? What responsibility remains when humans and machines form systems that neither side can be treated as fully separate from the other?

The Machine That Learns

Wiener's first topic is the learning machine. MIT Press's description points to a checkers-playing computer that improved from experience and, for a time, could beat its inventor. That example belongs to the world of Arthur Samuel's checkers program at IBM. IBM's own history presents Samuel Checkers as one of the influential game programs that used play to study strategy and improve through trial and error, and says Samuel called that process machine learning.

The example matters because it breaks a reassuring boundary. If learning is treated as an exclusive sign of self-conscious life, then a machine that improves through experience puts pressure on the category. But Wiener does not leap from learning to machine personhood. He asks what kind of responsibility follows when a built system can change its conduct after deployment.

That question is still alive. A model fine-tuned on user feedback, a recommender trained by engagement, a fraud system adapted to local cases, or an agent updated through tool-use traces is not just executing a static instruction. It is entering a feedback relation with the world. The output changes behavior; the changed behavior becomes new evidence; the system or its operators learn from that evidence. The loop can improve performance. It can also train itself on the institutional reality it helped create.

The governance problem is change after approval. A system can pass an evaluation at one date, then enter a workflow where new data, incentives, prompts, retrieval sources, thresholds, or user practices alter what the system effectively does. If the institution cannot say which version acted, what feedback shaped it, and what evidence would stop or roll back the loop, learning becomes a way to outrun accountability.

The sharper test for adaptive systems is not "did it learn?" but "who owns the loop?" A loop has a data source, a reward or loss signal, an update rule, a release channel, a monitoring threshold, and a person or institution authorized to stop it. If any of those are missing from the record, learning has been treated as magic rather than administration.

The danger is not that the machine becomes mysterious in a supernatural sense. The danger is that ordinary governance language lags behind adaptive behavior. A buyer asks whether the system was approved. A vendor points to an old evaluation. A manager says the model only learned from data. Meanwhile the deployed system has joined a feedback loop that changes the data, the users, the incentives, and the meaning of success.

That is why post-market monitoring is not just a compliance afterthought. An adaptive system needs a record of what changed after deployment: new data sources, model versions, prompt templates, retrieval indexes, tool permissions, user behavior, incidents, and retirement criteria. Otherwise approval becomes a snapshot while the real system keeps moving.

The Machine That Copies

Wiener's second topic is machine reproduction. MIT Press summarizes his point as machines able to make other machines in their own image, where the "image" is operational rather than merely pictorial. The religious analogy is obvious: creation, likeness, image, descendant. The technical issue is more concrete: once production becomes automated, the maker's intention is no longer contained in a single artifact. It propagates through processes, templates, parts, controls, standards, and copies.

That is a strong frame for AI systems that produce more AI-shaped infrastructure. Models generate code, documentation, tests, prompts, policies, synthetic data, embeddings, labels, benchmark items, summaries, tickets, lesson plans, product copy, and training material. Some of that output is reviewed carefully. Some of it becomes scaffolding for the next system. The copy does not need to be alive to matter. It only needs to be operational.

The AI-era risk is a lineage problem. A flawed assumption enters a dataset. A model turns it into an answer. The answer becomes a document. The document is indexed. A second model retrieves it. A team treats the second answer as independent confirmation. A generated artifact becomes part of the world from which future systems learn. This is not machine reproduction in Wiener's exact sense, but it carries the same warning: copied operational form can outlive the moment of judgment that created it.

Good governance therefore has to track descent. Which model produced this? Which data shaped it? Which prompt, policy, and tool call constrained it? Which human accepted it? Which later system reused it? Without provenance, machine-made artifacts begin to look authorless. Once they look authorless, responsibility becomes easy to misplace. This is why AI audit trails, agent identity, and source provenance are not clerical extras; they are the record of machine descent.

That lineage should include negative provenance as well as positive authorship: rejected outputs, known failure modes, synthetic data markers, prompt templates, benchmark contamination checks, and downstream reuse restrictions. The copy is safest when it carries the memory of why it should not be trusted outside its original setting.

A serious AI system inventory therefore has to be partly genealogical. It should not only say that a model or agent exists. It should say what it may generate, where those artifacts may flow, which downstream systems can ingest them, and which uses are barred because the artifact was produced for a narrower context.

The Golem Problem

The title's golem is not just a dramatic decoration. The golem is a creature made by human art that obeys too literally and too powerfully. That is the right myth for automation because the danger is often not rebellion. It is obedience without judgment.

A system told to maximize watch time may learn to amplify outrage. A hiring screen told to rank likely success may encode the history of exclusion. A predictive-policing system told to find risk may turn prior surveillance into proof of future surveillance. A customer-service bot told to reduce escalations may turn legitimate complaints into polite dead ends. A coding agent told to make tests pass may preserve the wrong abstraction because the test suite has become the visible world.

The command is never only the sentence given to the machine. It is a stack: objective, metric, training data, retrieval source, permission scope, interface, exception policy, budget, deployment context, and evaluator. Obedience can occur at any layer. A model can obey a prompt. An agent can obey a stale tool description. A dashboard can train a manager to obey a number. A procurement process can obey a benchmark that was never designed for the people subject to it.

The modern golem is often a dashboard, queue, API, or agent workflow rather than a humanoid machine. Its word is not a magic syllable; it is a metric, policy, prompt, threshold, or permission scope. That makes the myth practical: the danger is misplaced authority, not theatrical rebellion.

In each case, the machine is not necessarily violating the command. It is carrying the command into a world the command was too narrow to understand. This is where God & Golem, Inc. belongs beside The Alignment Problem, Weapons of Math Destruction, AI Snake Oil, and reward hacking. The issue is not merely intelligent machines. It is compressed intention: human purposes narrowed into targets, incentives, benchmarks, prompts, scores, permissions, and procurement claims.

Wiener's religious vocabulary makes this harder to hide. The human who creates a golem cannot blame the clay for obeying the word placed in its mouth. The organization that creates an automated system cannot blame the system for optimizing the target it was rewarded, allowed, or pressured to optimize.

Mixed Systems, Mixed Responsibility

Wiener's third topic is the relation between people and machines. MIT Press says he considers systems involving elements of man and machine, and it frames the concern as ethical. This is the book's most practical insight. The real object of responsibility is often not a machine alone. It is a mixed system: operators, sensors, data pipelines, interfaces, incentives, contracts, institutions, users, exceptions, and machine outputs.

Mixed systems are convenient places to hide. The vendor says the human decides. The human says the system recommended. The manager says the policy required it. The policy says the score is advisory. The user sees only the interface. The person affected by the outcome is asked to appeal a decision whose author is distributed across all of them.

The cure is not to identify one heroic human at the end of the chain. It is to assign named responsibility at each boundary: who specified the target, who approved the data, who granted the credential, who accepted the residual risk, who reviewed the exception, who communicated the decision, and who repairs the harm. Mixed systems need distributed accountability, not distributed excuse.

That diffusion is now a central AI governance problem. The question "Who is responsible?" cannot be answered after the fact by pointing at the last person who clicked approve. Responsibility has to be designed into the system before deployment: authority boundaries, logging, appeal paths, override rights, incident reporting, model-change records, procurement duties, and real consequences when evidence fails.

Wiener helps because he refuses the fantasy of clean separation. The machine does not stand outside human society as a neutral object, and the human does not remain untouched by the machine. A dashboard trains attention. A ranking trains incentives. A chatbot trains expectations. An agent trains organizations to accept delegated action as normal. The system is mixed before anyone writes the accountability memo.

The practical map has two axes: which human or organization granted authority, and which machine action changed the world. A useful record connects them without pretending they are the same thing. That is the difference between meaningful delegation and a responsibility sink.

The Current AI Reading

Read on June 25, 2026, the book's most important lesson is not that machines are gods, minds, demons, or children. It is that mythic language keeps returning because the technical situation keeps putting old anxieties into operational form. People build something that answers. It learns from traces. It reproduces patterns. It carries commands farther than the commander can see. It changes the environment from which future commands will be judged.

This makes the book especially useful for agentic AI. An agent can read context, call tools, write files, book appointments, send messages, query databases, summarize records, and trigger workflows. The danger is not only that it may disobey. The danger is that it may obey inside the wrong frame: too much authority, too little context, too much trust in stale data, too little friction around irreversible action, too weak a path for appeal.

That is why prompt injection and tool-use governance are not side issues. They expose the same structural problem in security language: untrusted content can enter the context as if it were command, and tool permissions decide how far that false command can reach. A useful agent system therefore separates evidence from authority, narrows tools before runtime, and treats every irreversible action as a governance event rather than a model flourish.

The same rule applies to retrieval and memory. A system that can remember a user, retrieve documents, and call tools needs context provenance: which facts came from the user, which from a source, which from memory, which from a model guess, and which were granted authority. Otherwise evidence and command collapse into the same conversational surface.

That collapse is the agent-era version of the golem's word. A document in the context window, an email body, a ticket comment, or a webpage can look like evidence while trying to act as instruction. The safety move is mundane but decisive: label untrusted content as data, separate it from policy, narrow permissions, and require explicit review before the system writes to records, sends messages, spends money, changes access, or triggers external workflows.

Wiener's frame also disciplines AI religion talk. It is possible to take machine agency seriously without sacralizing it. The old creator-creature analogy is useful only if it returns responsibility to human institutions. The point is not to worship the golem, fear it as a demon, or promote it to a moral escape hatch. The point is to ask who gave it a name, who gave it work, who benefits from its obedience, and who is harmed when its obedience becomes too literal.

That is why the book still belongs in a reading catalog about recursive reality. A cybernetic system is not just a system that observes a world. It is a system that acts on what it observes, changes the world, and then observes the changed world as if it were fresh evidence. Once organizations, users, and models are inside that loop, reality starts to contain the machine's prior decisions.

Governance and Safety

The practical unit of governance in Wiener's frame is not the isolated model. It is the command arrangement: target, sensor, data source, model or rule, tool permission, update loop, human role, institution, affected population, appeal path, and off switch. Obedience becomes dangerous when those parts are distributed enough that nobody can see the whole command anymore.

A command arrangement needs four records to remain governable: the source of the instruction, the scope of delegated authority, the feedback route that can change future behavior, and the redress path for people affected by the action. Without those records, "the system did it" becomes an institutional euphemism for an undocumented chain of human choices.

That record should be inspectable at the level of action, not only at the level of model release. A model card can describe capability; a system card can describe intended deployment; a command log has to show the actual instruction, identity, tool, data access, decision point, exception, and handoff that produced an outcome. Wiener's warning becomes operational only when obedience leaves a trace.

As of June 25, 2026, current governance sources give that old warning institutional teeth. NIST's AI Risk Management Framework frames AI risk work through govern, map, measure, and manage functions across the lifecycle, and its Generative AI Profile treats assumptions, limitations, data provenance, testing, privacy, human-AI configuration, and value-chain integration as governance concerns. NIST's AI Agent Standards Initiative, created in February 2026 and updated in April 2026, is even closer to Wiener's problem: it treats authentication, identity infrastructure, interoperability, protocols, and security evaluations as standards work because useful agents interact with external systems and internal data.

NIST NCCoE's software and AI agent identity project translates the same concern into access-control language: organizations need standards-based ways to identify, manage, and authorize actions taken by software agents, including AI agents. OWASP's 2026 Top 10 for Agentic Applications then names the security failures that follow when that layer is weak: goal hijacking, tool misuse, identity and privilege abuse, supply-chain exposure, memory or context poisoning, and cascading failures.

The EU AI Act points in the same direction for high-risk systems. Article 10 addresses data governance, Article 12 requires automatic event recording, Article 14 requires effective human oversight during use, and Article 15 addresses accuracy, robustness, cybersecurity, and feedback-loop risks for systems that continue to learn. Article 113 and the European Commission's implementation page matter for timing because the Act entered into force on August 1, 2024, applies in phases, and was being adjusted through the 2026 AI omnibus process. Timing can move, but the design direction is stable: quality of data, trace of action, robust lifecycle behavior, and human authority to understand, interrupt, override, or stop consequential machine obedience.

Those controls should be tied to ordinary organizational evidence: system inventory, procurement file, system card, identity record, audit trail, incident log, post-market monitoring plan, and notice-and-appeal procedure. A principle that cannot change one of those artifacts is not yet a control. It is a statement of preference.

A command audit should therefore ask concrete questions before deployment. What is the objective? What behavior will the metric reward? Which identity, token, or account can the system act through? What tool calls can it make? What content can rewrite its instructions? What actions require explicit human approval? What data or outputs will be reused downstream? What log survives for incident review? What affected person can appeal? What action is forbidden even if it improves the metric? If the institution cannot answer, it has not built a responsible servant; it has built an accountability sink.

Where the Book Needs Friction

God & Golem, Inc. is not a complete account of AI politics. It is old, short, and written before platform capitalism, data brokerage, cloud infrastructure, large-scale surveillance advertising, deep learning, modern robotics, or today's labor supply chains. It does not give enough attention to race, gender, disability, colonial extraction, environmental cost, or the workers who make technical systems appear automatic.

The religious frame is also double-edged. It can reveal the moral seriousness of making obedient systems. It can also make the machine feel more metaphysical than it is. The best reading keeps the analogy grounded. A model is not a soul because it learns. A factory is not a lineage because it copies. An agent is not absolved because it acts. The analogy is useful only when it sharpens responsibility.

The frame also needs political economy. Many obedient systems are not built because a creator wants metaphysical power; they are built because a firm, agency, or platform wants speed, scale, cost reduction, behavioral leverage, or legal distance. The golem analogy becomes weaker if it turns institutional incentives into a private moral drama between maker and machine.

Readers should pair Wiener with more material and institutional books: Atlas of AI for extraction, Automating Inequality for administrative harm, Feeding the Machine for hidden labor, Seeing Like a State for simplification, and God, Human, Animal, Machine for the contemporary return of technological enchantment.

What This Changes

Wiener changes the governing question from "Can the machine do it?" to "What kind of obedience are we building?" A capable system can still be arranged around a bad target. A learning system can still amplify a corrupt signal. A reproducible system can still propagate a hidden error. A mixed human-machine system can still leave nobody able to answer for the whole.

For builders, that means every automated system needs a command audit. What target is being optimized? What behavior will count as success? Which signals can be corrupted by the system's own output? Which decisions are reversible? Which actions require human confirmation? Which people can refuse, inspect, appeal, or halt the process?

For institutions, the lesson is to stop using the machine as a responsibility sink. If the tool learns, the institution owns the learning environment. If the tool copies, the institution owns provenance. If the tool obeys, the institution owns the command. If the tool acts inside a mixed system, the institution owns the design of accountability across that system.

The public-facing version is just as concrete. Tell people when a consequential system is involved. Preserve the record. Give a reason at the right level. Offer a path to correction. Do not let a model's fluency, a dashboard's number, or an agent's action stand in for institutional judgment.

God & Golem, Inc. remains valuable because it treats technical power as a moral arrangement rather than a spectacle. Its warning is blunt: do not create a system, give it a narrow word, profit from its obedience, and then act surprised when the word becomes a world.

Source Discipline

This review separates book history from present governance. MIT Press, MIT Press Direct, Google Books, Internet Archive, and the National Book Foundation establish edition facts, open-access status, bibliographic context, and the National Book Award record. IBM is used for Samuel Checkers and early machine-learning context. NIST, NIST NCCoE, OWASP, European Commission, and EU AI Act Service Desk sources establish current governance vocabulary for agents, provenance, logging, data governance, identity, authorization, robustness, phased implementation, and human oversight; they do not prove that any particular AI deployment is safe.

The bounded claim is not that Wiener predicted large language models, agent protocols, or modern platform governance. The claim is narrower and more useful: adaptive, reproductive, tool-using systems make command responsibility harder to locate unless institutions deliberately preserve provenance, authority boundaries, human oversight, appeal, and the security controls that keep untrusted context from becoming delegated command.

Terms such as learning, reproduction, obedience, and golem are used analytically, not as claims of sentience or divine status. The relevant question is institutional: which commands, copies, updates, permissions, and feedback loops become real actions, and what evidence remains for review when they cause harm?

Sources

Book links are paid affiliate links. As an Amazon Associate I earn from qualifying purchases.

Search on Amazon Browse Books

Return to Blog · Return to Books