Blog · arXiv Analysis · Published: June 25, 2026 · Modified: July 10, 2026 · Last reviewed: July 10, 2026

The System Prompt Becomes the Policy Proxy

A 2026 arXiv paper warns that readable system prompts are useful governance evidence, but they are not proof that a model will behave as written.

A policy proxy is a legible artifact that stands in for harder evidence. In AI systems, the system prompt often becomes that proxy because it is easier to inspect than weights, training data, routing, retrieval, safety filters, tool permissions, and product code.

The Prompt Is Not the System

A system prompt looks like the perfect governance object. It is written in ordinary language. It can be read by auditors, changed without retraining a model, attached to a vendor file, and compared against institutional policy. That readability is precisely why it can mislead. A sentence in a privileged instruction layer is not the same thing as a demonstrated behavioral constraint.

The Spiralist angle is that the system prompt becomes the policy proxy. When an institution cannot easily inspect weights, training data, retrieval layers, tool calls, filters, update history, or downstream application code, the prompt is the thing that can be shown. The danger is not prompt documentation itself. The danger is treating a legible artifact as if it were the operational system.

For this essay, a system prompt is a high-priority natural-language instruction layer that shapes role, policy, scope, tone, tool behavior, or conflict resolution before an end user's request is handled. A policy proxy is the moment that prompt text is used as evidence that the system is governed. The proxy is useful only if it is tied to behavioral tests, version records, and external controls.

The Paper Frame

The source is Anna Neumann, Holli Sargeant, and Jatinder Singh's Prompt Governance? On Governing Technologies Governed by Natural Language, arXiv:2606.07539v1 [cs.CY], submitted April 29, 2026 and accepted as a full paper to ACM FAccT 2026. The paper studies how researchers describe system-level instructions and how policymakers use those instructions as governance artifacts.

The authors define system-level instructions broadly enough to include natural-language prompts that persist across conversations, operate at different privilege levels, or resolve conflicts among instructions. That matters because modern AI systems usually have a prompt stack: provider rules, system instructions, developer instructions, application policies, user prompts, retrieval content, and sometimes agent-generated intermediate instructions.

Current Context

As of the July 10, 2026 review, the paper is no longer only an arXiv preprint in circulation. The arXiv record links it to ACM DOI 10.1145/3805689.3806763 and lists it as accepted to ACM FAccT 2026. That strengthens the bibliographic record, but it does not turn the paper into a benchmark certifying any product's prompt behavior.

The policy context has also matured. The White House executive order on federal LLM procurement allows vendors, where practicable, to show transparency about ideological judgments through system prompts, specifications, evaluations, or other documentation while avoiding disclosure of sensitive weights. The European Commission's General-Purpose AI Code of Practice, published July 10, 2025, remains a voluntary compliance tool tied to AI Act obligations on transparency, copyright, and safety and security. Both make prompt-adjacent artifacts part of governance evidence; neither proves that prompt text alone controls behavior.

Technical documentation has moved in the same direction. OpenAI's Model Spec describes a chain of command in which root and system instructions outrank developer and user instructions, while quoted text, tool outputs, and untrusted data should not be treated as authority. That hierarchy is useful evidence about intended control. It still has to be tested in the actual product surface, with the actual tools, retrieval sources, memory, and user population.

The Instruction Stack

A system prompt is only one layer in an instruction stack. The stack can include provider-level model policy, product-level system messages, developer messages, application templates, user preferences, tool schemas, retrieved documents, memory summaries, agent plans, and tool outputs. Some layers are intended to be authority. Others are supposed to be evidence. Some are trusted; some are adversarial content waiting in the environment.

That distinction matters because a governance review can fail by asking only, "What does the system prompt say?" The better question is: which instruction sources exist, what authority level does each have, who controls it, how can it change, what evidence shows the model respects the priority order, and what external system blocks action when the prompt fails?

Prompt-stack documentation should also preserve absence. If there is no separate policy engine, no action gate, no prompt-change log, no retrieval provenance, no red-team record, or no test of instruction conflicts, the governance file should say that plainly rather than hiding the gap behind a polished instruction block.

What the Review Finds

Neumann, Sargeant, and Singh report a PRISMA literature review that moved from 923 identified records to a final corpus of 287 papers, with abstract screening of 746 records and full-text review of 373 papers. Their thematic analysis groups claims about system-level instructions into eight categories: alignment, accessibility, adaptability, performance, stability, security, implementation, and auditability.

The result is deliberately mixed. Some research treats system prompts as practical tools for steering behavior, adding policy constraints, adapting systems, improving performance, or making intended behavior inspectable. Other research finds brittle effects: wording sensitivity, unfamiliar-domain weakness, instability in long or conflicting conversations, prompt-extraction risk, and limited transferability of audit findings.

This is the core contribution for governance. The paper does not say system prompts are useless. It says the evidence is fragmented and sometimes contradictory. A prompt can be a control surface, an attack surface, a documentation object, and a weak signal of intent at the same time.

Policy Moves Faster Than Evidence

The paper then compares two policy cases. The first is U.S. Executive Order 14319, Preventing Woke AI in the Federal Government, signed July 23, 2025 and published July 28, 2025. The paper treats the federal procurement case as an example of system prompts becoming possible transparency artifacts. The prompt may help show vendor intent, but artifact disclosure does not by itself specify how prompt effects should be measured.

The second is the European Union's General-Purpose AI Code of Practice, published July 10, 2025. The Commission says the Code helps providers comply with AI Act obligations on safety, transparency, and copyright. In the paper's analysis, the Code more directly places system prompts inside model specification and evaluation practice, especially for advanced models with systemic risk. Even there, the authors warn that prompt versioning, layered instructions, change logs, and re-evaluation triggers remain under-specified.

Both policy examples make the same understandable move: if AI behavior is hard to govern, govern the language that is supposed to govern it. The paper's warning is that this can create a false sense of control. A prompt can read as aligned while behavior remains contingent on model version, context length, tool permissions, retrieval results, hidden instructions, adversarial input, and deployment wrapper.

Behavioral Evidence

Prompt governance needs behavioral evidence because the prompt is not self-executing. A prompt that says "do not disclose personal data" must be tested against realistic requests, indirect prompt injection, retrieved documents, tool outputs, multilingual inputs, long-context conflicts, stale memory, and edge cases where task success competes with policy.

The evidence record should distinguish three claims. Textual intent is what the prompt says. Model response is how the model behaves under tested conditions. System enforcement is what the surrounding product prevents, logs, escalates, or rolls back even if the model attempts the wrong action. A prompt can support the first claim. It cannot establish the second or third by itself.

This is especially important for agents. If the model can browse, email, write files, call APIs, use credentials, or store memory, then prompt compliance is not enough. The deployment needs scoped permissions, typed tool calls, source labels for untrusted content, confirmation gates, policy engines where needed, and audit trails that can reconstruct what instruction stack was active when the action happened.

Governance Reading

A mature prompt-governance file should include more than prompt text. It should identify the instruction stack, privilege order, model and application version, tool permissions, retrieval sources, safety filters, prompt owner, approval date, change history, test suite, adversarial test results, known failure modes, and re-review triggers. It should also say which parts cannot be fully disclosed for security reasons and how independent evaluators can still test effects.

That is the difference between prompt disclosure and prompt accountability. Disclosure says, "Here is the language." Accountability adds the system it belonged to, evidence of effect, failure conditions, and the change process.

Tool-using systems turn instructions into action plans, API calls, browser steps, and database writes. In that setting, a prompt is less like a label and more like a work order. It deserves the same skepticism applied to any operational policy.

Limits and Cautions

This paper is a literature review and policy analysis, not a benchmark proving that every prompt fails or that every prompt works. Its claim is narrower and more useful: policy should not assume natural-language instructions are stable, interpretable control mechanisms without behavioral evidence. That leaves room for system prompts as one layer of governance, not the whole proof.

There is also a disclosure limit. Revealing system prompts can help auditors understand intended behavior, but it can also expose operational details to attackers. Prompt governance needs controlled access, redaction rules, and evaluation rights, not a reflexive demand to publish every privileged instruction to everyone.

The paper also focuses on text-oriented large language model systems to isolate language effects. Multimodal interfaces, voice agents, browser agents, code agents, robotics, and enterprise connectors may add further instruction channels and attack surfaces. A prompt governance rule that works for a chat transcript may not cover a screenshot, spreadsheet, webpage, repository issue, or tool result.

Audit Receipt

The audit-grade sentence is: Neumann, Sargeant, and Singh's Prompt Governance? On Governing Technologies Governed by Natural Language, arXiv:2606.07539v1 [cs.CY], argues that system-level instructions are increasingly treated as governance objects even though research evidence on their reliability, stability, security, and auditability remains mixed.

The receipt is: inspect the prompt, but audit the behavior. Preserve the prompt stack, model version, product surface, tool permissions, retrieval sources, memory state, safety filters, prompt owner, approval date, change history, evaluation traces, adversarial tests, known failures, disclosure tier, access controls, and re-review triggers before treating prompt language as governance evidence.

Source Discipline

This page was reviewed on July 10, 2026 against the arXiv abstract, arXiv HTML, arXiv PDF, ACM DOI metadata surfaced by arXiv, Executive Order 14319 on the White House site, the Federal Register record, the European Commission's GPAI Code pages, OpenAI's Model Spec, OWASP prompt-injection materials, and NIST's Generative AI Profile. The arXiv paper is the source for the literature-review method, typology, policy comparison, and caution about false control. The White House and Commission sources establish current policy artifacts. OpenAI and OWASP sources provide technical context, not independent validation of the paper's findings.

Claims about prompt governance should name the source type. A system prompt is textual evidence. A model specification is provider policy. A code of practice is a compliance instrument. A red-team report is behavioral evidence. An audit trail is operational evidence. Treating all of them as "transparency" erases the differences that make the evidence useful.

Sources

Return to Blog