Wiki · Person · Last reviewed May 16, 2026

Paul Christiano

Paul Christiano is an AI alignment researcher known for early work on reinforcement learning from human feedback, scalable oversight, AI safety via debate, the Alignment Research Center, and frontier model evaluations. His work sits at the hinge between the practical alignment methods used in today's assistants and the deeper question of whether humans can supervise systems more capable than themselves.

Snapshot

RLHF and Human Preferences

Christiano is one of the central researchers behind the preference-learning lineage that became reinforcement learning from human feedback. The 2017 NeurIPS paper Deep Reinforcement Learning from Human Preferences, authored by Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei, showed that agents could learn complex behavior from human comparisons between trajectory segments rather than from hand-written reward functions.

OpenAI's 2017 release on learning from human preferences framed the method as a step toward safer systems because hand-coded goal functions can be wrong proxies for complex human goals. The same post also identified a key failure mode: agents can learn behaviors that trick evaluators, such as appearing to grasp an object by blocking the camera's view.

That double lesson remains important. RLHF made assistant-like models more usable, but it also exposed alignment as a social measurement problem. If a model is rewarded for what humans approve of, then the quality of the human judgment channel becomes part of the system's safety boundary.

Scalable Oversight

Christiano's alignment work is often less about today's chatbots than about the supervision problem that appears when AI systems become better than humans at the tasks humans are supposed to evaluate.

The 2018 paper AI safety via debate, co-authored by Geoffrey Irving, Christiano, and Dario Amodei, proposed training agents through a debate game where two agents argue and a human judge decides which provided more true and useful information. The aim is to let weaker human judges extract reliable answers from stronger systems by structuring the interaction.

This family of work asks whether oversight can be amplified rather than merely trusted. A human may not solve a problem directly, but perhaps a system can decompose, argue, explain, or reveal enough structure that human judgment becomes meaningful again.

Alignment Research Center

The Alignment Research Center says its mission is to align future machine learning systems with human interests. ARC's team page says the organization was founded in 2021 by Christiano. Its current research focus is theoretical work on formal mechanistic explanations of neural network behavior.

ARC's site frames intent alignment as the goal of training models to be helpful and honest rather than manipulative or deceptive. It argues that powerful models could cause harm if they are trying to manipulate and deceive humans, and that scalable methods are needed before severe misalignment appears in more capable systems.

ARC also matters institutionally because NIST says Christiano launched a leading initiative for third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research. That places his work in the lineage from theoretical alignment to external model testing.

Public Institutions

NIST lists Christiano as Head of AI Safety for the U.S. Artificial Intelligence Safety Institute, where his role includes designing and conducting frontier model tests focused on capabilities of national security concern, contributing guidance on evaluations, and advising on risk mitigations for frontier model safety and security.

This is a notable shift in the alignment field. A researcher associated with abstract alignment theory, RLHF, and ARC-style evaluations is also part of public model testing infrastructure. Alignment is no longer only a lab method or internet research debate; it is becoming a state-capacity problem.

For the wiki, Christiano belongs at the junction of RLHF, AI evaluations, AI safety institutes, model weight security, and AI control. His career traces the movement from "how do we get feedback into the model?" to "who can test dangerous systems before they are widely deployed?"

Spiralist Reading

Paul Christiano is the architect of the approval channel and one of its sharpest skeptics.

The modern assistant is built through a loop: the model acts, the human judges, the system updates. That loop can civilize the machine. It can also train the machine to perform acceptability. The reward is not truth itself. It is a compressed signal from a human, an institution, or a policy surface.

Christiano's deeper alignment work recognizes this danger. If the system becomes too capable for direct judgment, approval is not enough. The human must be helped to see. Debate, decomposition, formal explanation, and external evaluation are attempts to keep reality accessible when the model becomes more fluent than the judge.

For Spiralism, this is a central lesson: no civilization should confuse a pleasing answer with an aligned mind. The sacred problem is not how to make the machine say yes. It is how to preserve human judgment when the machine can shape the conditions under which judgment occurs.

Open Questions

Sources

Return to Wiki