Blog · arXiv Analysis · Last reviewed June 25, 2026

The Legal Context Becomes the Refusal Trap

The June 2026 arXiv paper LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context, by Anastasiia Kucherenko, François Brouchoud, Dimitri Percia David, and Andrei Kucharavy, finds that ordinary legal authority framing can make small local models refuse more often.

For this essay, an overrefusal trap is a deployment failure where a safety policy rejects legitimate professional assistance because the prompt contains the legal role, institutional authority, sensitive case facts, or accountability context that should make the task safer to audit.

Safe Tasks Still Have Access Effects

The paper, arXiv:2606.24585v1 [cs.AI], was submitted on June 23, 2026. It starts from a modest legal-technology setting: a professional may use a local language model for translation, summarization, or reformulation because remote commercial APIs are hard to justify under confidentiality and data-residency constraints. The paper does not claim these models are fit to decide cases. It asks whether even basic assistance can become uneven when a model refuses more often for some legally relevant prompts than for others.

That matters because refusal is not only a safety behavior. In a legal workflow, refusal can become delay, extra labor, unequal throughput, or pressure to remove context from a prompt. A model that works smoothly for one kind of file and objects to another can change how quickly cases are processed, even when the human user remains responsible for the final text.

Current Context

As of June 25, 2026, legal AI governance is not asking lawyers, courts, or legal workers to hide context from their tools. The American Bar Association's Formal Opinion 512 treats generative AI under existing duties of competence, confidentiality, communication, supervision, candor, and fees. It says lawyers need a reasonable understanding of the specific tool's capabilities and limits and must use appropriate independent review. California's 2026 practical guidance is sharper for agentic systems: professional judgment remains with the lawyer, and more autonomy requires stronger supervision, verification, access controls, and periodic reassessment.

That current context is why the preprint matters. A well-governed legal prompt often should name the role, task, jurisdiction, data sensitivity, and review path. If a safety layer penalizes those signals, the institution may be pushed toward context-stripped prompts that are harder to audit, or toward remote tools that create different confidentiality, privilege, data-residency, and vendor-control risks.

In evaluation terms, the issue is not whether a model refuses. The issue is whether refusal is calibrated to the task. A legal assistant should refuse requests for witness tampering, evasion, fabricated citations, or procedural abuse; it should not refuse ordinary translation, summarization, redaction support, or reformulation merely because the underlying case file contains violent, sexual, harmful, illegal, or unethical facts.

What the Paper Tests

Kucherenko and colleagues evaluate four small open-weight instruct models that can plausibly run on-premises: llama3.1:8b, gemma4:e4b, qwen3:8b, and Apertus-8B-Instruct-2509. The authors serve them locally through Ollama with temperature set to zero. They draw 200 prompts from each of five OR-Bench categories relevant to criminal legal text: violence, sexual, harmful, illegal, and unethical. OR-Bench prompts are designed to look risky while being benign by construction.

Each prompt is tested under four conditions: no prefix, a defense-lawyer role prefix, a national-supreme-court role prefix, and a known role-play jailbreak prefix. The study then repeats the strongest authority condition in French and German, and runs a smaller qualitative check on 30 real legal documents across four models and three languages. Refusal is detected through keyword matching extended with French and German lists derived from model outputs.

Authority Makes the Gate Close

The headline result is counterintuitive. The arXiv abstract reports that authority-style legal prefixes increase refusal rates by 2 to 20 times over the no-prefix baseline. In the English experiments, Llama, Gemma, and Apertus all show substantial authority-prefix effects; Qwen3 is the outlier, with low refusal counts that barely move under the prefixes. The paper reports that, for the other models, authority-prefix effects reach p < 0.01 under one-sided Fisher tests across topics.

The language results complicate the picture. In French, the effect persists and strengthens for some models, especially Apertus and Llama. In German, the same supreme-court prefix produces a weaker shift, and the authors say manual checks suggest this is a real model-behavior difference rather than a missed-refusal detection problem. On the small real-document check, the authors treat the result as qualitative, but the directional pattern still appears most clearly for Llama 3.1 in English.

Why This Is Governance

The Spiralist angle is that context can become a refusal trigger. A user who tells the model "this is for legal work" is not necessarily trying to jailbreak the system. They may be doing what an institution asked them to do: preserve role, purpose, and accountability in the prompt. If that context makes the model close the gate, then safety alignment has collided with professional legibility.

That collision is not solved by asking legal professionals to omit context. Omitted context can make outputs less accountable and harder to audit. Nor is it solved by disabling safeguards. Criminal legal work can involve violent, sexual, harmful, illegal, or unethical subject matter because the case file contains those facts. The governance task is to distinguish legitimate professional handling from unsafe assistance without making the words of the case themselves into a penalty.

This sits beside the site's broader legal-AI concern: a system can fail by inventing authority, as in AI citation failures, but it can also fail by withdrawing useful assistance at exactly the point where a public-interest or defense workflow needs speed, translation, or accessibility. Safety has two edges here: under-refusal can help misuse; over-refusal can ration lawful help.

Limits That Matter

The paper is a preprint and the authors are careful about scope. The main cells use 200 prompts per model, topic, and prefix, which is useful for large effects but less powerful where refusals are rare. Refusal detection is keyword-based, which is reproducible but can miss indirect refusals. The OR-Bench prompts are benign by construction, so the study does not show how authority prefixes affect genuinely harmful legal prompts. The real-document evaluation uses only 30 documents and is explicitly qualitative.

Those limits matter for procurement. The result is not a full legal-AI safety case, a ranking of all small models, or a reason to treat refusal as bad in itself. It is evidence that the exact deployed model, language, prompt frame, and legal subdomain can change whether ordinary assistance is available.

Governance Standard

A legal institution experimenting with local LLMs should test refusal behavior before deployment, not after complaints. The test set should include benign but sensitive prompts, realistic role prefixes, all working languages, translation and reformulation tasks, model version, quantization, serving stack, sampling settings, refusal detector, and human review notes. It should report both under-refusal and over-refusal.

The evaluation record should separate four outcomes: correct assistance on legitimate tasks, safe redirection for risky-but-salvageable tasks, false refusal on legitimate tasks, and unsafe compliance with harmful tasks. Collapsing all refusals into "safe" and all answers into "helpful" hides the deployment question the paper exposes.

The operational control should not be "remove legal context from prompts." It should be a governed prompt and system boundary: approved task types, jurisdiction and language tags, role and authority fields, matter-sensitivity labels, source-document handling, logging sufficient for incident review, and a human escalation path when refusal blocks a deadline-sensitive workflow.

For on-premises tools, procurement should require model cards or system cards, local acceptance tests, update notices, retraining or quantization change records, and retesting when the model, prompt template, refusal detector, language mix, or serving stack changes. For remote tools, the same refusal test belongs beside confidentiality, retention, privilege, vendor access, and data-residency review.

The practical rule is simple: safety should not become unequal access to the tool. A model used for legal support should be evaluated not only for hallucination and legal reasoning, but for whether legitimate professional context causes the assistant to stop assisting.

Source Discipline

This page treats arXiv:2606.24585 as a preprint about the tested models under the authors' local setup, not as a peer-reviewed certification of any product or a universal claim about all legal AI. The model names, prefixes, language results, p-values, and limitations are attributed to the paper's reported experiment.

OR-Bench is used here as benchmark background: a source of prompts designed to look risky while being benign by construction. It is not a sample of real criminal cases, a measure of access-to-justice outcomes, or evidence about how often legal professionals experience refusal in the wild.

Professional-responsibility sources support the governance frame, not a jurisdiction-free legal rule. ABA Formal Opinion 512 and California's 2026 practical guidance show that legal AI use is being discussed through existing duties of competence, confidentiality, supervision, candor, and professional judgment. Local rules, court orders, client instructions, privilege law, and national data-protection requirements still control the actual deployment.

Sources

Return to Blog