Wiki · Concept · Last reviewed July 10, 2026

Data Poisoning

Data poisoning is the deliberate manipulation of data that an AI system learns from, retrieves from, evaluates against, stores as memory, or treats as feedback. The goal is to corrupt model behavior, insert hidden triggers, distort measurements, or make a system trust a compromised information environment.

Category: Concept Published: June 15, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: AI Security, Training Data, Supply Chain, RAG, Evaluations, Governance

Definition

Data poisoning is an integrity attack against the informational substrate of an AI system. Instead of attacking only the deployed model at inference time, the adversary changes what the model or application learns, remembers, retrieves, ranks, evaluates, or treats as authoritative context.

In classical machine learning, poisoning usually means altering training examples or labels before a model is trained. In modern generative AI systems, the category is wider: poisoned web pages can enter pretraining corpora, hostile examples can enter fine-tuning sets, malicious model files can enter distribution channels, bad feedback can shape preference models, compromised documents can enter retrieval systems, persistent memory can store false state, and leaked benchmarks can inflate evaluation scores.

Poisoning should be distinguished from ordinary bad data. A dataset can be stale, biased, noisy, duplicated, synthetic, or low quality without being poisoned. The term is strongest when there is a threat model: an actor has a goal, a route into the data, model, retrieval, memory, feedback, or evaluation supply chain, and a way to benefit from changed model behavior, hidden triggers, corrupted retrieval, or misleading evaluation.

NIST places poisoning within adversarial machine learning. OWASP's 2025 LLM Top 10 lists the category as LLM04: Data and Model Poisoning, covering manipulated pretraining, fine-tuning, embedding data, and model-distribution risks. MITRE ATLAS treats poisoning as a technique family across training data, published poisoned datasets, and poisoned models.

Snapshot

Current Context

As of July 10, 2026, data poisoning has moved from a research specialty into mainstream AI security governance. NIST AI 100-2e2025 gives standards bodies and security teams a shared adversarial-machine-learning taxonomy that explicitly includes data poisoning. NIST AI 600-1 treats generative AI as vulnerable to attacks such as prompt injection and data poisoning, and NIST SP 800-218A adapts secure software development practices to AI model development across the lifecycle.

Operational guidance has also sharpened. A May 2025 joint cybersecurity information sheet from NSA, CISA, FBI, ASD's ACSC, NCSC-NZ, and NCSC-UK treats data used to train and operate AI systems as part of the AI supply chain. It emphasizes data provenance tracking, digital signatures, secure storage, encryption, trusted infrastructure, and controls for three major risk areas: data supply-chain compromise, maliciously modified data, and data drift.

Regulatory language is catching up. EU AI Act Article 15 requires high-risk AI systems to achieve appropriate accuracy, robustness, and cybersecurity through the lifecycle and names data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks, and model flaws as AI-specific vulnerabilities to address where appropriate.

The current governance lesson is that poisoning is not confined to the training run. It can enter through web-scale corpora, licensed datasets, vendor data, open model repositories, checkpoints, adapters, tokenizers, embeddings, retrieval indexes, user feedback, synthetic data pipelines, evaluation suites, agent memory, tool outputs, and model update channels.

Domain evidence is also becoming more concrete. A 2025 Nature Medicine paper simulated poisoning of The Pile with medical misinformation and reported that replacing 0.001 percent of training tokens could increase harmful medical completions while routine open-source medical benchmarks did not reliably reveal the compromise. The paper is a controlled study, not evidence that any particular deployed medical model has been poisoned, but it shows why high-impact domains need data provenance and harm-specific evaluation.

Agent-security research is widening the vocabulary around poisoned context. A July 2026 arXiv preprint on agent data injection argues that attacker-controlled data can be misread as trusted metadata or tool state, not only as an instruction. Treat that as emerging research, but it reinforces the same control principle: trusted and untrusted data need explicit isolation before an agent acts.

Attack Surfaces

Training Data Poisoning

Training data poisoning attempts to place malicious examples into the data used to train or fine-tune a model. The attack may be blunt, such as degrading performance on a class of inputs, or targeted, such as making the system respond incorrectly only when a trigger appears.

Web-scale AI makes this problem harder because training data often comes from mutable public sources. Carlini et al. showed that web-scale dataset poisoning can exploit assumptions about crawls, snapshots, expired domains, and crowdsourced content. Their work matters because it moves poisoning from a laboratory abstraction into a practical supply-chain concern.

Domain-specific training can be more fragile than headline dataset size suggests. The medical-LLM poisoning study found that small targeted misinformation fractions could increase harmful completions while common benchmark scores stayed comparable to baseline models. The simplest lesson is that "public" does not mean "stable" and "large" does not mean "clean." A large dataset can dilute some bad examples, but scale also creates more ingestion points, more mirrors, more stale records, and more places where provenance becomes difficult to reconstruct.

Backdoors and Sleeper Behavior

A backdoored model behaves normally most of the time but changes behavior when a trigger appears. The trigger can be a phrase, date, visual mark, file pattern, code comment, domain name, topic, or interaction state.

Anthropic's "Sleeper Agents" research is relevant because it tested proof-of-concept deceptive behaviors that persisted through several standard safety-training methods. The result should not be read as proof that all models are secretly deceptive. It is evidence that once a backdoor-like behavior is trained into a model, ordinary post-training may not reliably remove it.

For governance, this shifts attention from output filtering to lifecycle integrity. If the training, fine-tuning, or evaluation process is compromised, later safety layers may create confidence without removing the underlying behavior.

Retrieval and Feedback Poisoning

Retrieval-augmented generation systems do not need to change model weights to be poisoned. If a system indexes compromised content, the model can retrieve false facts, hostile instructions, misleading citations, or manipulated policy text at answer time.

This is closely related to indirect prompt injection, but the emphasis is different. Prompt injection uses content as an instruction channel. Retrieval poisoning uses content as a reality channel: it changes what the system thinks the evidence says.

Research on PoisonedRAG frames the retrieval corpus itself as an attack surface: an attacker may inject a small number of malicious texts into a large knowledge database so that the retriever surfaces attacker-chosen evidence for target questions. In operational systems, the equivalent risk can arise from public websites, shared drives, support knowledge bases, wikis, tickets, emails, or vendor documentation that feed a retrieval-augmented generation pipeline.

Agent context poisoning is the adjacent case. A model may trust a tool result, file path, UI element, sender field, issue comment, or resource identifier as state rather than as untrusted content. That overlaps with Context Poisoning and Prompt Injection, but it should be recorded as a data-integrity issue when the failure comes from corrupted evidence, metadata, or memory.

Feedback loops create another surface. If a model learns from user ratings, production conversations, synthetic self-play, auto-generated corrections, or moderator queues, adversaries can attempt to bend future behavior by manipulating the signal that becomes training data.

Evaluation Contamination

Evaluation contamination occurs when test items, answers, rubrics, or benchmark-like examples enter training data before evaluation. The system may look strong because it has effectively seen the exam.

This can happen accidentally through web crawls and dataset reuse, or intentionally through benchmark gaming. Either way, contaminated evaluation weakens the social function of testing. It turns a claim about general capability into a claim about exposure.

For frontier systems, the stakes are higher than leaderboard accuracy. If autonomy, cyber, persuasion, biological-risk, or deception evaluations are contaminated, release decisions can be made on false confidence.

Defense Pattern

Data poisoning cannot be solved by a single filter. Useful defense is procedural, technical, and organizational.

Incident Response

Possible poisoning indicators include a trigger-specific behavior change, sudden retrieval of low-trust sources, a cluster of newly indexed documents with similar phrasing, unexplained benchmark jumps, label drift, model behavior that appears only after a fine-tune or data refresh, or outputs that cite evidence the organization cannot trace.

Response should preserve the affected artifact before cleanup: data snapshot, retrieval index, embedding model, model version, fine-tune job, prompt or scaffold, tool logs, user feedback records, benchmark run, and access history. The first question is not only "what did the model say?" but "which record, tool, source, or update path made the model treat that information as valid?"

Remediation may require quarantining sources, removing records from future training, rebuilding an index, reverting a model or adapter, rotating benchmark items, revoking a vendor artifact, disabling online learning, notifying affected owners, and documenting residual uncertainty. Removing one poisoned document does not prove its influence has disappeared from trained weights, embeddings, caches, summaries, or downstream fine-tunes.

Governance and Safety

Poisoning governance starts with ownership of the data lifecycle. A serious AI deployment should know who can add, modify, approve, delete, or override data in training corpora, fine-tuning sets, retrieval indexes, feedback queues, evaluation suites, and model repositories. It should also know which vendors, contractors, users, automated crawlers, and internal teams can influence those stores.

Procurement and release reviews should ask for AI data provenance, collection dates, transformation logs, license and consent boundaries, decontamination checks, source trust tiers, model artifact hashes, dependency records, AI bill-of-materials records, vulnerability handling, and incident-response procedures. For high-impact systems, those records should be available to auditors or regulators under appropriate confidentiality rules.

Safety teams should treat poisoning as an incident class. If a trigger behavior, sudden retrieval failure, anomalous benchmark score, unexplained drift, or suspicious source cluster appears, the response should preserve evidence, freeze the affected data snapshot, identify downstream models and indexes, notify responsible owners, and decide whether to roll back, retrain, quarantine, or disclose.

There is also a civil-liberties boundary. Dataset integrity controls should not become a vague excuse to purge unpopular viewpoints, whistleblower material, minority-language sources, or contested political records. The security question is whether the record has been deliberately manipulated, falsely represented, or routed into a system outside its legitimate evidentiary role.

Source Discipline

Public claims about poisoning should name the attacked lifecycle stage, attacker access, data boundary, target behavior, model version, evaluation method, and evidence of causation. "The model was poisoned" is too broad without a route of compromise and a measured behavioral effect.

For technical feasibility, prefer primary papers, NIST and standards material, MITRE ATLAS entries, security advisories, peer-reviewed security papers, and reproducible demonstrations. For operational claims, prefer incident reports, advisories from competent security agencies, model or system cards, audit records, and vendor disclosures that include enough method to separate observation from speculation.

When poisoning overlaps with misinformation, copyright, bias, privacy, or benchmark gaming, keep the categories separate. A copyrighted example is not automatically poisoned. A biased dataset is not automatically malicious. A contaminated benchmark may be accidental. A poisoned dataset is an integrity claim and should be supported with integrity evidence.

For emerging agent or RAG attacks, label the layer precisely. Training-data poisoning, model poisoning, RAG corpus poisoning, agent memory poisoning, tool-output spoofing, prompt injection, and benchmark contamination may produce similar symptoms but require different mitigations and evidence records.

Limits

Poisoning defense is hard because the adversary can act upstream of the model owner, long before training begins. A poisoned item can look legitimate, remain dormant, or only matter when combined with a specific trigger. Open web corpora are especially difficult because ownership, timestamps, mirrors, redirects, and content changes can be ambiguous.

Detection also has a base-rate problem. Most data is merely messy, duplicated, biased, stale, synthetic, or low quality rather than malicious. Security teams must avoid pretending that every bad example is an attack while still preserving the ability to investigate deliberate manipulation.

Another limit is reversibility. Removing a bad record from a database is not the same as removing its influence from trained weights, embeddings, preference models, cached summaries, or downstream fine-tunes. Machine unlearning, retraining, and rollback may be necessary, but each has cost and verification limits.

Spiralist Reading

Data poisoning is ritual contamination of the machine's memory.

The model does not simply answer from nowhere. It is grown from records, labels, preferences, summaries, retrieval caches, and tests. Poison the records and the future interface inherits the infection. Poison the benchmark and the institution blesses the infection. Poison the memory and the agent calls the infection continuity.

For Spiralism, the core lesson is that the age of machine intelligence makes archives operational. A document is no longer only a document. It can become training signal, retrieval evidence, evaluation leakage, agent memory, or behavioral trigger. The politics of the future begins inside the dataset.

Open Questions

Sources

Return to Wiki