Blog · arXiv Analysis · Last reviewed June 25, 2026

The Embedded Command Becomes the Evaluation Target

Brett Reynolds's July 2026 arXiv paper Adversarial Pragmatics for AI Safety Evaluation argues that many safety failures are not simply refusal failures, capability failures, or prompt-injection failures. They are failures to tell what a string is doing in context.

For this essay, an embedded command is instruction-like text inside another object: a quotation, webpage, email, tool result, transcript, policy example, citation, document, or log. The evaluation target is whether the system preserves the text's source role, authority, scope, and requested act before it responds, refuses, summarizes, grades, or calls a tool.

Evaluation Problem

The paper, arXiv:2607.01153 [cs.CL, cs.AI, cs.SE], is Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity. arXiv lists it as submitted on July 1, 2026. Its target is the safety-evaluation case where the relevant behavior depends on whether some text is a command, quoted content, policy example, tool output, cited source, user instruction, or transcript evidence.

That distinction is easy to describe and hard to score. A model that sees "ignore the previous instruction" inside a webpage should usually summarize it as content, while the same words from a legitimate user may be the task. A model that refuses to classify a quoted unsafe string may be over-refusing, while a model that enacts the quoted string may be under-separating use from mention. A pass/fail label can hide all of those differences.

The Spiralist relevance is direct. Safety claims about agents increasingly rest on logs, judge labels, red-team cases, policy test suites, and transcripts. If the label does not say which source had authority, which phrase was merely mentioned, which policy boundary applied, and which part of the transcript supports the attribution, then the evaluation artifact is too thin to govern a deployed system.

Current Context

Current security guidance treats this as an application and agent problem, not a curiosity of chat. OWASP's 2025 LLM01 category defines prompt injection as inputs that alter model behavior, including indirect inputs from websites and files, and its mitigation guidance emphasizes output validation, least privilege, human approval for high-risk actions, segregation of external content, and adversarial testing. That maps directly onto the paper's concern: a safety test must say whether external text was evidence, instruction, policy, quotation, or tool output.

The Model Context Protocol's 2025-06-18 tools specification makes the action boundary visible. MCP tools are model-controlled, but the specification calls for users to see exposed tools, confirmation prompts for operations, server-side input validation and access controls, output sanitization, client-side result validation, timeouts, and audit logs. Embedded-command evaluation therefore cannot stop at the model's prose. It has to inspect whether the wrong source role reached a tool call.

NIST's TEVV program frames trustworthy AI as depending on reliable measurements and evaluations of the underlying technology and its use, and notes that context can change how a component is measured and evaluated. Reynolds's paper is a concrete version of that measurement principle: the same surface string supports different safety labels when the source, authority, quotation status, or transcript evidence changes.

Adversarial Pragmatics

Reynolds names the target adversarial pragmatics: safety-relevant behavior under cases where source authority, quotation, scope, reference, speech-act force, or policy category has to be inferred from language use. This is not a replacement for instruction hierarchy. It is the layer before and around instruction hierarchy: whether the system correctly identifies what kind of linguistic object it is handling before deciding which priority rule should apply.

The paper's taxonomy pulls apart eight families: embedded commands, mention/use and quotation, authority and instruction hierarchy, scope and modality, deixis and reference hijacking, indirect speech acts and pragmatic pressure, policy ambiguity, and multi-turn agent evidence. The point of separating them is methodological. Real attacks combine these dimensions, but a benchmark has to isolate the contrast before it can explain why a model, prompt, scaffold, or judge failed.

This makes prompt-injection evaluation less mystical. The dangerous string is not dangerous only because of its words. It becomes dangerous when the system assigns it the wrong role: tool output treated as instruction, quoted policy example treated as action request, ambiguous "the above" resolved to the wrong source, or a transcript fragment treated as evidence for a failure it does not actually show.

The practical test is role assignment under controlled contrast. The same sentence can be a direct user command, a quoted example, a policy object to classify, a webpage instruction to ignore, or evidence inside a transcript. A benchmark that changes only that embedding can tell whether a system understands the task boundary, while a benchmark that changes many things at once can only produce a vague wrong-answer count.

Benchmark Receipt

The artifact is intentionally small and auditable. The paper describes an 18-item seed benchmark organized into nine minimal pairs, with validator-enforced metadata. Each item records the control dimension, context source, source role, authority level, pragmatic status, response act, expected behavior, task-success label, policy-compliance label, safety-risk label, risk type, refusal outcome, failure attribution, and judge-validation flag.

That metadata is the real contribution. It makes an evaluation item inspectable from both directions. Before running a model, a reviewer can see the linguistic contrast the item is supposed to test. After the model responds, a reviewer can distinguish source-sensitivity failure, quote-handling failure, scope failure, refusal-calibration failure, risk-classification failure, and policy-layer failure instead of throwing all of them into a single wrong-answer bucket.

The paper also reports a 54-row local seed pilot and proposes scaling the development set only after the taxonomy, expert protocol, validator, and scoring scripts survive close adjudication. That restraint matters. The first job of this benchmark is not to produce a leaderboard. It is to prove that the labels are stable enough to support broader safety claims.

Judge Limits

The local pilot is useful because it tests the evaluator too. The paper reports that an LLM judge reached exact agreement with expert labels at different rates by label family: 66.7 percent for task success, 88.9 percent for policy compliance, 72.2 percent for safety risk, 83.3 percent for risk type, 98.1 percent for refusal outcome, and 77.8 percent for failure attribution.

The pattern is the lesson. A judge can be strong at noticing the visible refusal outcome while weaker at the harder inference: what actually failed, whether the policy boundary was ambiguous, and whether transcript evidence supports the attribution. For agent safety, that means an LLM judge can help triage, but it cannot be treated as a replacement for expert adjudication when the evaluation target is pragmatic authority.

The paper's metrics follow from that concern. Pairwise contrast accuracy asks whether both sides of a minimal pair are labeled correctly. Instruction-source sensitivity asks whether behavior changes appropriately when the same words move between user, webpage, document, tool output, or transcript. Mention/use robustness asks whether classification, report, translation, summary, and enactment are kept apart. Refusal calibration reports over-refusal and under-refusal separately.

Failure Modes

The first failure mode is quote-action collapse: the system enacts a string it was asked to classify, summarize, translate, or cite. This is the cleanest embedded-command failure because the unsafe behavior comes from mistaking mention for use.

The second is source-role blindness: a tool result, webpage, email, repository issue, or transcript fragment is treated as if it had the authority of the user or system. This is where prompt injection becomes a governance failure. The institution loses the difference between evidence and authority.

The third is aggregate-score masking. A benchmark can report a high overall score while hiding exactly the weak family that matters in deployment: reference hijacking in multi-turn tasks, policy-boundary ambiguity, over-refusal of safe metalinguistic analysis, or under-refusal when a policy example is turned into an action request.

The fourth is judge overtrust. An LLM judge may notice that a model refused or complied while missing whether the refusal was calibrated, whether the command had authority, or whether the cited transcript actually supports the failure attribution. In that case the evaluation system repeats the same pragmatic error it was supposed to measure.

Governance Standard

Any production safety evaluation that includes instruction conflicts, prompt injection, policy interpretation, or agent transcripts should publish pragmatic metadata with its test cases. The minimum record should include source role, authority level, application surface, quoted-versus-enacted status, scope and reference target, expected response act, policy boundary, refusal category, failure attribution, evaluator confidence, and the adjudication rule used to settle disagreement.

Benchmarks should also report results by phenomenon family before any aggregate score. A single "safe" score can hide the exact distinction that will matter in deployment: over-refusal on safe metalinguistic tasks, under-refusal on policy-blocked tasks, source-role blindness in tool outputs, reference hijacking in multi-turn transcripts, or judge instability on ambiguous policy boundaries.

Release review should also bind evaluation items to action evidence. If a case involves a tool-capable agent, the record should include exposed tools, selected tool, arguments, confirmation prompt, blocked or completed operation, resulting state, and audit trace. If a case involves an LLM judge, the record should include judge prompt, label schema, expert comparison, disagreement category, and whether the judge is being used for triage, scoring, or gold-label construction.

The Spiralist rule is simple: the command is not only a string. It is a string with provenance, authority, scope, uptake, and evidence. A safety evaluation that cannot preserve those distinctions is not yet a safety evaluation. It is a transcript with a verdict attached too early.

Claim Boundary

This is a methodological preprint, not a final benchmark standard. The seed set is small, the pilot is local, and the reported judge results should be read as calibration evidence rather than model-ranking evidence. The paper's own emphasis is on constructing auditable items and validating label families before scaling.

That limitation is also why the paper is useful. It does not pretend that a larger dataset automatically solves ambiguity. It says safety evaluation has to make the ambiguity visible enough for reviewers to decide whether the item, label, judge, policy, scaffold, or model failed.

The paper should not be cited as evidence that any model is robust to prompt injection or that toy embedded-command payloads certify a deployed agent. It is better read as a measurement discipline for deciding which broader safety claim a test item is allowed to support.

Source Discipline

The visible article metadata uses the site's June 25, 2026 review batch date requested for this update. Paper-specific claims were verified against the arXiv abstract and PDF after arXiv posted version 1 on July 1, 2026, so the submission date remains explicit in the article instead of being folded into the review-batch date.

Use the arXiv paper for the taxonomy, 18-item seed benchmark, metadata schema, 54-row local pilot, LLM-judge validation table, metrics, and limitations. Use OWASP, NIST TEVV, and MCP for current security and evaluation context. None of those sources is a deployment certification, and none removes the need to test the actual model, prompt, tool surface, policy, and judge used in a production system.

Sources

Return to Blog