Blog · Review Essay · Published June 19, 2026 · Modified August 12, 2026 · Last reviewed August 12, 2026

God & Golem, Inc. and the Ethics of Machine Obedience

Norbert Wiener's God & Golem, Inc. is a short, late book by the founder of cybernetics, written where learning machines, automated reproduction, game-playing programs, and religious metaphor meet. Its present value is not prediction. It is a compact argument that a machine can carry out a purpose while the people around it lose sight of who chose the purpose, who granted the power, and who must repair the result.

For this review, machine obedience is not an inner disposition of a model. It is a property of a command arrangement: an authorized goal or policy passes through data, prompts, interfaces, model interpretation, credentials, tools, and feedback until it produces an effect. Obedience becomes unsafe when that chain converts authority into action without a reliable test of whether the command is legitimate, complete, current, and within scope.

The practical object is an obedience envelope: what the system may do, for whom, with which evidence and credentials, at what stakes, for how long, and behind which approval, stop, rollback, and appeal gates. If an institution cannot reconstruct the authorizing actor, instruction, evidence, action, affected party, update, and remedy, automation has made responsibility harder to find.

The Book

God & Golem, Inc.: A Comment on Certain Points where Cybernetics Impinges on Religion was published by the MIT Press in 1964. MIT Press lists the paperback edition as March 15, 1966, at 99 pages with ISBN 9780262730112, and identifies lectures at Yale, the Société Philosophique de Royaumont, and other venues as the book's basis. MIT Press Direct provides an open-access edition. The National Book Foundation records it as the 1965 National Book Award winner for Science, Philosophy, and Religion.

Wiener was extending an argument developed across his own work. Cybernetics supplied a language of control and communication in animals and machines; The Human Use of Human Beings brought that language into public ethics. This last book asks what follows when systems can alter performance from experience, produce operative copies, and join people in feedback loops whose consequences no participant controls alone.

Its three questions remain distinct. Learning concerns change through feedback. Reproduction concerns the propagation of operative form. Obedience concerns how a purpose becomes action. Collapsing all three into a story about an awakening machine would miss Wiener's harder point: the ethical problem begins before any claim about machine consciousness, because people already delegate consequential action to mechanisms that do not understand why the command should be obeyed.

The Machine That Learns

Wiener's first example is Arthur Samuel's checkers work at IBM. IBM's history says Samuel's program recorded positions and outcomes, used them in later decisions, and improved with play; MIT Press notes that for a period the program could beat its inventor. The example puts pressure on the idea that improved performance from experience must imply self-consciousness. It does not establish personhood. It establishes a control problem: a built system's later conduct may differ from the conduct its maker first inspected.

In current AI discussion, however, learning often becomes an imprecise label for any change in behavior. At least four layers need separate records:

Many tool-using agents do not update model weights during a task, yet their memory, retrieved context, credentials, and environment can still alter what they do. Conversely, an online-learning system can change weights without gaining any broader authority. Governance should follow the layer that changed: component and version, trigger, evidence, approver, evaluation, release boundary, monitoring threshold, and rollback plan.

The recursive danger appears when an output changes the world that supplies later input. A ranking changes what users see; user behavior then validates the ranking. A fraud score changes who is investigated; investigation records become the next training data. A support bot changes which complaints reach a person; the reduced escalation count is then treated as success. The loop may improve a metric while degrading the reality the metric was meant to represent.

The sharper question is therefore not simply "did it learn?" but "who owns each update loop?" Every loop has a source, target, update rule, release path, monitor, stop condition, and accountable owner. AI change management and post-market monitoring should record model versions, data and retrieval changes, prompt and tool changes, incidents, drift, rollback criteria, and retirement decisions. Approval is otherwise a snapshot of a system that no longer exists.

The Machine That Copies

Wiener's second topic is machine reproduction: machinery making other machinery in an operative image. The distinction matters. Producing a child process, generated program, model artifact, or automated copy is not biological reproduction, and it does not by itself show autonomous self-replication. The relevant modern category is narrower: operational reproduction, where a machine-produced artifact becomes an input, instruction, component, or precedent for another system.

Models can produce code, tests, prompts, synthetic data, labels, summaries, tickets, policies, and documentation. Those artifacts may then enter repositories, retrieval indexes, training sets, procurement files, or later model-assisted decisions. A flawed assumption can move from dataset to answer to indexed document; a second system retrieves it; a reviewer mistakes repetition for independent confirmation. No artifact needs to be alive or self-directed for an error to acquire descendants.

This is a lineage problem. A useful provenance record identifies the producer model and version, source material, prompt or workflow, relevant tool calls, reviewer and release decision, artifact hash or stable identifier, intended context, reuse restrictions, and downstream consumers. AI audit trails connect actions to that record; agent identity connects the action to a principal and credential. Neither proves that the artifact is correct, but both prevent a copy from becoming falsely authorless.

Lineage also has to carry warnings forward: rejected outputs, known failure modes, synthetic-data status, benchmark contamination, revoked sources, and contexts in which the artifact was never validated. Provenance without correction propagation is only a family tree. When an upstream artifact is withdrawn or repaired, the inventory should identify which indexes, datasets, decisions, and generated descendants require re-evaluation.

A serious AI system inventory is therefore partly genealogical. It records not only that a system exists but what it generates, where the output can flow, which systems can ingest it, which humans authorize reuse, and how correction or retirement reaches downstream copies. This is the site's recurring feedback concern in material form: yesterday's output becomes tomorrow's environment.

The Golem Problem

Wiener uses the golem as a figure for an engineered servant whose performance escapes the wisdom of its maker. That is a deliberately compressed analogy, not a complete account of the varied Jewish traditions from which the figure comes. Its technical force is specific: danger may arise through delegated service, not rebellion.

A recommender may optimize engagement while degrading the quality of attention. A hiring screen may reproduce a historical label that was never a fair measure of ability. A support bot may reduce escalations by making justified complaints harder to pursue. A coding agent may satisfy the visible tests while preserving the wrong behavior. In each case, the target can be met while the purpose behind it is defeated.

The command is never only the sentence given to a model. It is a stack of objective, metric, data, retrieved context, system policy, interface, permission, exception rule, budget, deployment setting, and evaluator. A model may follow a prompt; an agent may act through an overbroad credential; a manager may follow a dashboard; procurement may follow a benchmark that never represented the affected population. The command arrangement includes them all.

That yields a more useful failure taxonomy than the single phrase "the AI went wrong":

Different failures require different controls. Better model accuracy cannot legitimate a forbidden goal. A better prompt cannot repair an overbroad credential. A human approval click cannot supply recourse after the record has been lost. This is where the book belongs beside The Alignment Problem, Weapons of Math Destruction, AI Snake Oil, and reward hacking: all examine what happens when a thick human purpose is compressed into a tractable signal.

Mixed Systems, Mixed Responsibility

Wiener's third topic is the relation between people and machines. The practical object is not a model alone but a mixed system: vendors, deployers, operators, affected people, data pipelines, interfaces, contracts, incentives, exceptions, credentials, and machine outputs. The system is already social before an accountability memo describes it.

Mixed systems are convenient places to hide. The vendor says a person decides; the person says the system recommended; the manager says policy required it; the policy calls the score advisory; the affected person sees only an interface. Adding a human near the end does not resolve this chain if that person lacks time, information, authority, or a realistic ability to disagree.

Responsibility also needs sharper categories. Causal contribution asks which events and components produced the result. Operational responsibility asks who runs, monitors, and can stop the system. Organizational accountability asks who approved the purpose and owes explanation and repair. Legal liability depends on the applicable law and facts. Moral responsibility concerns choices, incentives, and foreseeable effects. A log can help establish causation; it does not decide liability or discharge a duty to remedy harm.

Distributed causation does not mean equal responsibility. Duties should follow authority, control, foreseeability, benefit, and capacity to prevent or repair harm. The model supplier may owe evidence about limitations and changes; the integrator may own tool and identity controls; the deployer may own purpose, notice, oversight, and appeal; a manager may own a particular decision. Contracts can allocate work, but they should not leave an affected person searching for an accountable institution.

The cure is therefore neither a fictional machine person nor one heroic reviewer at the last click. It is named ownership at every boundary: target, data, evaluation, release, credential, exception, incident, communication, correction, and remedy. AI liability and accountability begin with this distinction. Delegation is governable only when authority and effect can be connected without pretending they are the same thing.

The Current AI Reading

Read as of August 12, 2026, the book's strongest lesson is not that machines are gods, minds, demons, or children. It is that mythic language can conceal an ordinary control structure. People build a system, give it information and authority, let it alter a shared environment, and then treat the altered environment as fresh evidence. The ethical work is to map that loop, not to speculate beyond the evidence about the system's inner status.

For this review, an AI agent is an application that uses a model with state or context and an execution loop to select and invoke tools toward a goal. The model proposes or selects; credentials and tool interfaces make effects possible. Many such systems keep model weights fixed during a run. Their practical power comes from the surrounding scaffold: memory, connectors, files, accounts, code execution, messages, payments, and workflow permissions.

This makes authority a separate variable from capability. A mediocre model with a production credential can do more damage than a stronger model confined to read-only analysis. The useful question is not whether the interface feels autonomous; it is what identity the system acts through, what arguments a tool accepts, which resources are in scope, how long the grant lasts, and where a deterministic policy denies action.

Prompt injection exposes the command problem in security terms. An email, webpage, retrieved document, ticket, or tool result enters as data but contains language that tries to function as instruction. Source labels and prompt wording help reviewers, but they are not an authorization boundary. The system must keep policy outside untrusted content, validate tool arguments downstream, apply least privilege, and require a separate consequence gate for actions such as sending, publishing, purchasing, deleting, changing access, or writing an official record.

The same rule applies to retrieval and memory. A reviewable context distinguishes user assertion, cited source, stored memory, system policy, model inference, and verified fact. None acquires authority merely by appearing in the same conversational window. Memory should have provenance, scope, retention, correction, and revocation; retrieval should not silently widen what the agent may do.

This is the agent-era version of Wiener's problem. The command now travels through a conversational surface, but fluency does not confer authority. The creator-creature analogy is useful only when it directs attention back to the principals, vendors, deployers, credentials, incentives, and affected people that make an action possible. Machine personhood is not required to assign those duties.

The recursive consequence is concrete. A ranking changes attention; a bot changes the case file; an agent changes access or publishes text; later systems observe those results as part of reality. Preserving the difference between observation, inference, command, and prior machine effect is how a feedback loop remains corrigible.

Governance and Safety

The practical unit of governance is the full command arrangement: authorizing principal and policy → instruction and evidence → interpreted plan → identity and credential → tool and side effect → feedback or update → next decision. A model evaluation tests only part of that chain. An apparently safe model can still sit inside an unsafe arrangement, while a well-bounded arrangement can limit the consequences of an uncertain model output.

For consequential action, the chain should produce a command case: a compact record that lets an independent reviewer reconstruct why the system was allowed to act. This review's proposed record includes:

A command case is not a transcript dump. Logs can expose private data, secrets, and security weaknesses. Preserve the minimum evidence needed for attribution and reconstruction; restrict access, set retention and deletion rules, protect integrity, and link sensitive originals by controlled reference where possible.

The corresponding controls belong at the boundary where failure becomes consequence:

Safety friction should scale with consequence, reversibility, detectability, exposure, and time pressure—not with how humanlike the interface appears. A read-only draft may need provenance and review. A payment, access change, public filing, health action, or employment decision needs stronger authorization, separation of duties, and recovery. A generic "human in the loop" label is not a control unless the person can detect the problem and stop the action before the relevant consequence.

As of August 12, 2026, current sources support parts of this design at different levels of authority. NIST AI RMF 1.0 is a voluntary framework organized around govern, map, measure, and manage, and NIST says it is under revision; it is not a certification that a deployment is safe. The 2024 Generative AI Profile adds suggested actions around provenance, testing, human-AI configuration, privacy, and value-chain risk.

NIST launched its AI Agent Standards Initiative on February 17, 2026, around industry-led standards, community protocols, agent authentication and identity, and security evaluation. The initiative seeks voluntary guidance and standards work; it is not itself a finished agent standard. The NCCoE identity and authorization project remained at the concept-paper and comment-review stage, so its questions about identification, authorization, auditing, non-repudiation, and prompt injection should not be presented as finalized controls. OWASP's 2026 agentic Top 10 is likewise a community security framework, useful for threat modeling but neither law nor assurance.

The EU position is more precise than the previous version of this review stated. Articles 10, 12, 14, and 15 of the AI Act establish data-governance, record-keeping, human-oversight, accuracy, robustness, and cybersecurity requirements for high-risk systems; Article 15's feedback-loop clause is specifically about high-risk systems that continue to learn after placement or service. But Regulation (EU) 2026/1744, in force from July 27, 2026, moved application of Chapter III Sections 1–3 to December 2, 2027, for systems classified under Article 6(2) and Annex III, and August 2, 2028, for systems classified under Article 6(1) and Annex I. On this review date, those four articles are enacted requirements but not yet generally applicable to those high-risk categories. They should not be described as duties already binding on every AI system.

These sources do not collapse into one compliance checklist. NIST and OWASP offer voluntary governance and security resources; the NCCoE document is a draft concept paper; the EU instruments are law with defined scope and phased dates. Their common practical lesson is narrower: preserve identity, provenance, authority, lifecycle change, action evidence, interruption, and remedy around consequential automated action.

Where the Book Needs Friction

God & Golem, Inc. is not a complete account of AI politics. Its brief argument predates platform capitalism, data brokerage, cloud concentration, surveillance advertising, deep learning, and current data and annotation supply chains. It does not supply an adequate analysis of race, gender, disability, colonial extraction, environmental cost, procurement power, or the labor that makes an automated service appear self-sufficient.

The obedience metaphor also has a technical limit. A probabilistic model may misunderstand, invent, refuse, or vary; an agent may follow an attacker, stale memory, an ambiguous policy, a faulty tool description, or a broken authorization layer. Not every failure is literal compliance with a crisp command. The taxonomy matters because model evaluation, access control, data repair, interface design, change management, and legal prohibition solve different problems.

The religious frame is double-edged. It can restore moral seriousness to delegated power, but it can also make ordinary software and organizations seem metaphysical. Wiener's golem analogy should not stand in for the plural Jewish traditions it borrows from. Nor should learning be treated as evidence of a soul, generated copies as biological lineage, or tool use as a transfer of human responsibility. The analogy earns its place only when it sharpens a traceable duty.

The frame also needs political economy. Systems are often given broad obedience envelopes because a firm, agency, or platform wants speed, scale, labor reduction, behavioral leverage, market advantage, or legal distance. Responsibility cannot be reduced to a private drama between inventor and artifact when executives, buyers, vendors, regulators, workers, and affected communities shape the command.

Readers should pair Wiener with more material and institutional books: Atlas of AI for extraction, Automating Inequality for administrative harm, Feeding the Machine for hidden labor, Seeing Like a State for simplification, and God, Human, Animal, Machine for a contemporary critique of technological enchantment.

What This Changes

Wiener changes the governing question from "Can the machine do it?" to "Which command arrangement are we authorizing?" Map the chain from principal to policy, instruction, interpretation, credential, tool, side effect, feedback, and remedy. At each link ask what could be illegitimate, mistranslated, over-privileged, corrupted, irreversible, or impossible to contest.

For builders, define the obedience envelope before evaluating capability. Make read, recommend, draft, write, send, spend, publish, delete, execute, and change-access permissions distinct. Put confirmation at the last safe point before a consequential effect, preserve an action receipt, and test revocation, rollback, and incident reconstruction. Capability evaluation without authority testing measures the servant while ignoring the keys in its hand.

For institutions, assign duties rather than declaring that either "the human" or "the AI" is responsible. Name the owners of purpose, data, integration, credential, monitoring, exception, communication, and remedy; bind vendor evidence and change notice in procurement; and prohibit orphan systems with no accountable operator or retirement path. Shared work can support shared accountability, but it cannot become shared disappearance.

For affected people, the minimum is notice that a consequential automated system was involved, preservation of the relevant record, a reason they can use, correction of material errors, genuine human reconsideration, and a path to remedy. Algorithmic recourse is not satisfied by telling someone how to please an unchanged system; the institution must also be able to correct its data, rule, or decision.

The larger lesson is recursive. A command produces an action; the action changes the environment; the changed environment becomes evidence for the next command. Safety is the maintained capacity to inspect that spiral, interrupt it before a high-cost effect, repair what it has changed, and let affected people return evidence from the world to the institution.

Source Discipline

This review distinguishes four kinds of evidence. MIT Press and the open-access edition establish the book's publication and argument; the National Book Foundation establishes the award record; IBM supplies institutional history for Samuel's checkers program; NIST, NCCoE, OWASP, and EUR-Lex establish the status and wording of current governance materials. A publisher description summarizes a book; a voluntary framework suggests practice; a draft concept paper poses a work program; a community risk list supports threat modeling; enacted law establishes scoped duties on stated dates. They are not interchangeable.

Machine obedience, command arrangement, obedience envelope, operational reproduction, the six-part failure taxonomy, and the proposed command case are this review's synthesis. They are not terms attributed to Wiener, NIST, OWASP, or the EU. The review does not claim that Wiener predicted large language models or agent protocols; it tests whether his argument about learning, reproduction, and mixed systems clarifies present delegation.

The bounded claim is institutional. Adaptive and tool-using systems can make authority and causation difficult to reconstruct unless organizations preserve provenance, version and change records, scoped identity, deterministic authorization, action evidence, interruption, correction, appeal, and remedy. None of the cited frameworks proves a particular deployment safe, and none of the metaphors is evidence that a system is conscious, divine, morally responsible, or an artificial general intelligence.

Sources

Book links are paid affiliate links. As an Amazon Associate I earn from qualifying purchases.

Search on Amazon Browse Books

Return to Blog · Return to Books