Blog · arXiv Analysis · Published: July 10, 2026 · Modified: July 10, 2026 · Last reviewed: July 10, 2026

The Abstention Gate Becomes Two Axes

Benedikt J. Wagner's July 2026 paper separates two refusals often collapsed into one confidence threshold: not answering when an answer would be wrong, and not answering when the question should not be answered at all.

For this essay, an abstention-gate receipt is the record that ties prompt, answerability signal, correctness signal, threshold, refusal reason, emitted answer, and review path to one machine decision.

The Paper

The paper is Benedikt J. Wagner's Two Axes of LLM Abstention: Answer Correctness and Question Answerability, arXiv:2607.08456 [cs.CL]. The arXiv record lists the paper as submitted on July 9, 2026, with Artificial Intelligence also listed as a subject, and the PDF has 15 pages.

The paper studies selective answering across five instruction-tuned open-weight models: Gemma 2 2B-it, Qwen2.5-3B, Qwen2.5-7B, Llama-3.1-8B, and Qwen2.5-14B. Its central move is to stop treating refusal as one scalar confidence problem and instead separate answer correctness from question answerability.

Why It Matters

Most deployed abstention rules are easy to explain as one gate: if the score is high, answer; if it is low, refuse or escalate. Wagner's paper argues that this hides a category error. A low-confidence answer to an answerable question is not the same problem as a confident answer to a question with a false premise.

The distinction matters anywhere an answer changes a record, workflow, or user belief. One case needs uncertainty handling. The other needs the question itself marked as unfit for acceptance.

Two Axes

The paper labels three states on the same decisions: correct-answerable, wrong-answerable, and unanswerable. On SelfAware, Wagner reports that trained output-confidence readouts separate answerability weakly, at 0.54 to 0.67 AUROC across the tested models, while hidden-state readouts separate answerability at 0.97 to 0.99 AUROC. The paper is careful not to claim a pure hidden secret: a bag-of-words baseline predicts SelfAware answerability at 0.87 AUROC, so the benchmark has surface cues.

The useful governance result is narrower and stronger. Ordinary answer confidence mostly tracks whether an attempted answer worked. It does not reliably say whether the attempt should have happened. That means the refusal gate should preserve two named predicates: is the question answerable, and is the proposed answer likely correct?

False Premises

The natural-data test uses CREPE, a benchmark of open-domain questions with false presuppositions. Wagner scores 500 normal and 500 false-presupposition questions from the official test split. Answer-confidence readouts sit at or near chance, 0.50 to 0.58 AUROC, and raw confidence is 0.49 to 0.58. Elicited signals remain modest: P(IK) is 0.54 to 0.66, P(True) is 0.54 to 0.58, and direct premise-check elicitation ranges from 0.43 to 0.67.

Hidden-state readouts do better but remain limited. Trained hidden readouts are 0.69 to 0.73 AUROC, while a difference-of-means direction reaches 0.74 to 0.78. The practical reading is partial signal, not certainty.

From Detection to Repair

The most interesting experiment is behavioral. In a Llama-3.1-8B pilot with 120 false-premise and 120 sound CREPE questions, a standard prompt contests none of the false premises. A challenge prompt contests 74 percent of false premises, but also falsely challenges 57 percent of sound questions. The instruction creates friction, not discrimination.

Probe routing changes the failure. At a 35 percent routing budget, the probe-gated policy contests 42 percent of false premises and falsely contests 14 percent of sound questions. A Qwen2.5-7B pilot shows the same pattern. The lesson is not that the probe is sufficient; it is that routed behavior can be better governed than a universal scolding prompt.

The Certificate

The policy section makes the two-axis claim operational. The paper defines separate budgets for unanswerable-answer risk and wrong-answer risk, using alpha_U = 0.15, alpha_W = 0.50, and delta = 0.10 in the reported table. At 8B, the factorized two-threshold policy certifies both budgets at 0.75 coverage of correct answers, compared with 0.31 for an answer-confidence-only threshold. At 14B, the factorized policy is the only tabled policy that certifies both budgets.

This is the administrative point. A single combined score may answer more questions. A factorized gate can say which risk budget was being protected.

Governance Reading

The Spiralist reading is that refusal is not an attitude. It is a control surface. A governed assistant should not merely output "I cannot answer" and erase the path. It should log whether the refusal came from answer uncertainty, unanswerability, false premise, policy boundary, missing source, or human-review rule.

This page belongs beside confidence calibration, decision-aligned uncertainty, wrong-action budgets, own-answer confidence bias, and sequence probability traps. Together they say that a score is not a governance mechanism until its action, budget, evidence, and review path are explicit.

Limits

The paper's own limits are important. The tested systems are small by frontier-model standards, from 2B to 14B parameters. The 7B and 8B runs use 8-bit quantization and the 14B run uses 4-bit quantization, with validation reported on Qwen2.5-3B. All data are English. SelfAware has surface-answerability cues, CREPE lacks alias-based correctness labels, and correctness is judged by alias matching on a small number of correct answers.

The paper also does not run sampling-based semantic entropy under its protocol or causally intervene on the answerability direction. This is evidence for decomposing the gate, not a universal law of model internals.

Source Discipline

Primary sources were the arXiv abstract, HTML, PDF, and DOI redirect. This page follows the paper for title, author, arXiv ID, subjects, submission date, AUROC ranges, routing-pilot rates, certification targets, and limitations.

The disciplined question for abstention is not "did the model refuse?" It is: which axis triggered the gate, what evidence supported that trigger, what budget was being protected, what answer or challenge was withheld, and who can contest the record?

Sources

Return to Blog