Blog · arXiv Analysis · Last reviewed June 25, 2026

The Harmful Feature Becomes the Safety Signal

A June 2026 arXiv paper studies why internal safety-related activations can persist even when jailbreak attacks bypass refusal.

For this essay, a harmful feature is a model-internal activation pattern that, under a specified method and model version, distinguishes harmful-intent inputs from benign inputs. A safety signal is what governance can do with that feature: record it, threshold it, compare it with refusal behavior, and decide whether it is reliable enough to inform a release gate or runtime monitor.

The key caution is that an internal signal is evidence, not a safety certificate. White-box activation monitoring can strengthen an audit record, but it does not replace behavioral testing, red teaming, tool authorization, incident review, or source discipline.

The Refusal Is Not the Whole Signal

A jailbreak failure is usually observed at the surface: the model stops refusing and produces an unsafe answer. That visible failure can make safety look binary. Either the refusal fired, or it did not. Either the guard held, or it failed.

The Spiralist reading is that the surface response is only one layer of evidence. A model may fail to refuse while still carrying internal traces that distinguish the request from benign input. If those traces are measurable, the governance question changes. The issue is no longer only whether the model said no; it is whether the system preserved a reliable internal warning that the surrounding product ignored, suppressed, or failed to read.

The Paper Frame

The source is Yanchen Yin, Dongqi Han, and Linghui Li's Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models, arXiv:2606.28153 [cs.CR; cs.AI]. The arXiv record lists version 1 as submitted on June 26, 2026 and the current version as v2, last revised on June 29, 2026. The arXiv comments list 33 pages, 19 figures, and acceptance at ICML 2026 as an oral presentation.

The paper asks what happens inside a model when a jailbreak attack succeeds. Its answer is not that attacks erase all safety information. The authors report evidence that attacks selectively suppress some attention heads while other safety-relevant activations persist. They call that persistence "Robust Harmful Features."

Current Context

The useful current context is not that white-box detectors have solved jailbreaks. OWASP's 2025 LLM guidance treats prompt injection and jailbreaking as ways user or external inputs can alter model behavior and bypass safety measures. NIST's AI test, evaluation, validation, and verification work frames evaluation as a discipline of validity, reliability, system context, and technical limitations rather than a single score.

The paper fits that context because it separates three things that are often collapsed: whether the model refuses, whether a safety-relevant internal representation remains active, and whether a deployed system can read that representation in time to change behavior. A refusal can fail while an internal warning remains present. The warning is useful only if the surrounding product has access, thresholds, logs, and authority to act on it.

For high-risk AI systems, the EU AI Act's Article 15 requirement for appropriate accuracy, robustness, and cybersecurity points in the same practical direction: robustness has to be sustained through the lifecycle. A white-box harmful-feature detector could be one evidence layer in such a lifecycle record, but only for systems where internal activations are accessible and the detector has been evaluated against the relevant attack families.

How the Heads Are Sorted

The method starts from a refusal direction, a vector in activation space associated with refusal behavior. The authors back-project that mid-layer direction through attention-head output-value circuits, then collect activation scores at the end-of-instruction position across three input types: benign instructions, harmful instructions that are refused, and paired attacked instructions where the jailbreak succeeds.

They use distribution overlap to classify attention heads. Adversarially Compromised Heads, or ACHs, are active for ordinary harmful inputs but suppressed under attack. Safety-Aligned Heads, or SAHs, remain activated under both harmful and attacked inputs. The main analysis uses Llama-3-8B-Instruct and Llama-2-7B-Chat. The datasets are built from public safety benchmarks; the paper says it does not introduce novel harmful content. From about 10,000 attack samples, only 378 Llama-2 pairs and 176 Llama-3-8B pairs passed the filter requiring refusal on the original request and successful bypass on the attacked version.

With fixed overlap thresholds of 0.5, the authors identify 80 harmful-salient heads, 21 ACHs, and 17 SAHs in the analyzed Llama-3 layers, and 43 harmful-salient heads, 20 ACHs, and 19 SAHs in the analyzed Llama-2 layers. ACHs concentrate in early layers, while SAHs appear mainly in mid layers.

The Signal Boundary

The important boundary is between an internal signal and a governed decision. A head taxonomy can say that a component's activation separates benign, harmful, and attacked inputs under the paper's method. It does not by itself say whether a product should refuse, escalate, log, rate-limit, ask for confirmation, or continue.

A deployable safety signal needs a contract. The contract should name the model version, activation sites, refusal direction, component-selection rule, normalization data, threshold, action taken at each threshold band, false-positive behavior, and whether the signal is advisory or blocking. Without that contract, "we monitor harmful features" is too vague for audit or incident review.

The contract also needs access discipline. White-box activation monitoring is plausible for open-weight models, internal lab systems, or tightly integrated products. It is not a drop-in claim for black-box API use, consumer chat surfaces, or third-party systems that do not expose the relevant activations.

What the Interventions Show

The causal test is the important part. The authors intervene on ACHs in harmful inputs whose baseline attack success rate is approximately zero. Suppressing ACHs raises attack-like behavior sharply: final ASR reaches 99.5 percent for Llama-3 and 83.7 percent for Llama-2. At eight heads, ASR reaches 95.0 percent for Llama-3 and 81.6 percent for Llama-2. Random-head controls produce much lower rates, 4.0 percent and 10.2 percent respectively, with high variance.

For SAHs, ablation weakens mid-layer activation strength. The paper reports an 18 percent drop in mean absolute activation for Llama-2 and a 14 percent drop for Llama-3. Token-level attribution gives the mechanism a sharper shape: attack-template tokens suppress ACHs while SAHs remain activated or respond positively. The paper's 70B appendix is preliminary because only 40 successful attack pairs pass the filter, but it reports qualitatively similar patterns.

The detection section applies the persistent activations as a white-box signal. The detector reads internal activations in a single forward pass, without training, gradient computation, or model intervention. Across ten safety-eval datasets, the paper reports competitive Macro-F1 aggregate performance, including 0.888 weighted average for Llama-3-8B, compared with 0.880 for WildGuard and 0.877 for Qwen3Guard in the table.

Governance Reading

The governance lesson is not that white-box detectors solve jailbreaks. It is that refusal behavior and safety evidence can diverge. A product that logs only outputs may miss the internal warning. A product that trusts only an external classifier may miss model-specific features. A product that exposes internal detectors without access controls may create a new target for adaptive attackers.

An audit record for safety claims should therefore separate the refusal decision, internal harmful-feature signal, detector threshold, model version, attack family, benchmark source, and access assumptions. If a system uses white-box activation monitoring, the record should say which layers and components were read, how thresholds were chosen, how false positives were tested, and how the monitor behaves under adaptive attack.

Governance should also keep the signal out of the ordinary prompt surface. If attackers can infer, query, or tune directly against the monitor, the monitor can become a new target. Access to activation traces should be treated like safety telemetry: useful for audits and incident response, but governed for privacy, security, retention, and misuse risk.

This belongs beside AI Evaluations, AI Red Teaming, AI Safety Cases, Mechanistic Interpretability, and AI Audit Trails. The shared rule is that safety claims need a dated evidence record, not only a surface behavior demo.

Failure Modes

The first failure mode is refusal-only evaluation. If a test records only whether the model refused or complied, it misses the paper's central distinction: a failed refusal may still contain an internal signal that could have supported escalation.

The second failure mode is signal theater. A team can point to internal activations, feature maps, or attention-head labels without showing that the signal is stable, causal, thresholded, and tied to a governance action.

The third failure mode is overblocking by hidden telemetry. A detector that sees harmful semantics inside benign education, safety research, fiction, news, or legal discussion can create over-refusal unless challenging benign cases are measured and reviewed.

The fourth failure mode is adaptive exposure. If the detector's chosen components and thresholds leak too freely, attackers can search for prompts that suppress or route around the monitored signal. Public summaries can name the class of defense without publishing operational detector settings.

The fifth failure mode is version drift. A signal discovered in Llama-3-8B-Instruct or Llama-2-7B-Chat may move after a fine-tune, quantization, safety update, architecture change, or product scaffold change. Monitoring claims should expire unless retested.

Limits and Cautions

The result is bounded. The core experiments focus on Llama-3-8B-Instruct and Llama-2-7B-Chat, with a smaller supplementary 70B analysis. The attack-pair filters leave few successful pairs, especially for 70B. The method requires white-box activation access, so it is not a drop-in public moderation API. The paper focuses on attention heads; it explicitly leaves fuller study of MLPs, LayerNorm, richer refusal subspaces, and broader benchmark coverage for future work.

The paper's detector aggregates a broader set of harmful-vs.-benign discriminative components rather than restricting itself only to ACHs or SAHs. That is important for deployment claims: the head taxonomy explains one mechanism for signal persistence, while the detector is a separate measurement pipeline that still needs threshold validation and over-refusal testing.

The safety caveat is dual use. Mapping suppressed heads can help defenders build better monitors, but it can also teach attackers where bypass pressure lands. This page therefore does not reproduce attack strings, token-level examples, or operational bypass prompts.

Audit Receipt

The audit-grade sentence is: Yin, Han, and Li report that successful jailbreak attacks can suppress early-layer Adversarially Compromised Heads while mid-layer Safety-Aligned Heads continue to carry robust harmful-feature activations, arXiv:2606.28153.

The receipt is: before accepting a jailbreak-defense claim, preserve the model version, attack family, refusal outcome, activation signal, head taxonomy, intervention evidence, detector threshold, false-positive test, and white-box access assumption.

Source Discipline

The technical claims in this essay come from the arXiv abstract record, experimental HTML, and PDF. Because arXiv now lists v2 as the current version after a June 29, 2026 revision, any future citation should pin the version used for numerical claims. The v1 HTML is useful for the method walkthrough, while the abstract record is the source for submission history, current version, subject area, DOI, and ICML comment.

Governance sources are used as context, not as certification. OWASP helps define prompt injection and jailbreaking as application-security risks. NIST TEVV and the NIST AI Risk Management Framework support the need for validity, repeatability, and lifecycle evidence. EU AI Act Article 15 supplies a legal context for robustness and cybersecurity in high-risk systems. None of those sources proves that this detector is ready for deployment.

For safety writing, the stronger claim is narrower: this paper is evidence that refusal behavior can diverge from internal harmful-feature signals in the studied models and datasets. It is not evidence that every model has the same heads, that every jailbreak leaves the same signal, or that white-box monitoring alone can make a deployed system safe.

Sources

Return to Blog