Blog · arXiv Analysis · Last reviewed June 25, 2026

The Circuit Map Becomes the Variance Problem

The June 2026 arXiv paper Demystifying Variance in Circuit Discovery of LLMs, by Frank Zhengqing Wu, Francesco Tonin, and Volkan Cevher, asks whether a discovered circuit is stable enough to govern with. Its answer is uncomfortable: a circuit map is not a permanent wiring diagram. It is a method-conditioned hypothesis that can change when the data batch changes, when the prompt template changes, or when the metric is read sample by sample.

The Map Moves

The paper, arXiv:2606.16920 [cs.LG], was submitted on June 15, 2026. It studies circuit discovery, a mechanistic-interpretability practice that tries to identify the model components most important for a task. The target is not interpretability in general. It is the reliability of the circuit-discovery procedure itself.

Wu, Tonin, and Cevher focus on EAP-IG, an edge-attribution patching method based on integrated gradients. The arXiv abstract reports that EAP-IG performs well on the usual faithfulness and unfaithfulness metrics, but suffers from three kinds of variability: resampling variance, rephrasing variance, and sample-wise variance.

Variance here has a precise governance meaning. It is the instability of the discovered subgraph under legitimate changes to sampling, wording, or metric aggregation. A circuit claim therefore needs an uncertainty record in the same way an evaluation claim needs a test protocol. The earlier CircuitLasso essay asked whether circuit discovery can become cheap enough to repeat. This paper asks what happens when repetition returns different maps.

Current Context

As of June 25, 2026, mechanistic interpretability is increasingly discussed as part of AI audits, safety cases, and evaluation records. That raises the evidentiary bar. A circuit diagram can support an assurance argument only if the report says what system, task distribution, prompt family, method, graph-size rule, and validation metric produced it.

NIST's AI Risk Management Framework is voluntary, but it frames trustworthy AI as a design, development, use, and evaluation problem, not a one-time disclosure. NIST's TEVV work is even more direct: reliable AI depends on measurement and evaluation, and context affects how each AI characteristic is measured. Circuit discovery should inherit that discipline. A circuit map is an evaluation artifact with scope conditions, not a free-standing proof of internal truth.

This matters because interpretability can otherwise become more persuasive than it is warranted to be. A graph of edges looks causal, but the paper shows that the graph may depend on the batch, the wording, or a metric denominator. The responsible current posture is to treat circuit maps as bounded evidence, tied to model versions and task distributions, and to combine them with behavioral evaluations, red-team findings, monitoring, and change-control records.

What CEAP Fixes

The authors introduce CEAP, conductance-based edge attribution patching. Their theoretical claim is narrow: conductance satisfies additive order preservation, a scoring property they argue integrated gradients lacks. The empirical claim is also narrow. CEAP lowers resampling variance, measured by higher or comparable pairwise Jaccard overlap across circuits discovered from different samples of the same distribution, while keeping unfaithfulness comparable to EAP-IG.

The experimental setup matters. The paper tests GPT-2 small, GPT-2 XL, Pythia-160M, and Pythia-2.8B on SVA, IOI, and greater-than tasks. For each task it uses roughly 10,000 samples, draws four 1,000-sample subsets, discovers circuits, and compares graph overlap.

For governance, CEAP is not a magic stabilizer. The appendix says CEAP clearly improves or matches pairwise Jaccard overlap but does not consistently beat EAP-IG on unfaithfulness. Its useful contribution is procedural: if a circuit map changes because the batch changed, the method can make that instability smaller and more measurable, while preserving a record of where it remains uncertain.

Template Variance

The harder result is rephrasing variance. The paper shows that prompts human reviewers would treat as the same task can activate different circuits when written in different templates. In the authors' terms, different templates trigger different circuits inside the model.

This is a direct challenge to a common interpretability dream: find the task circuit, steer it, and get reliable control. If a task can be expressed in many templates, and unseen templates use different internal routes, then a circuit found on the available templates may steer a benchmark without steering the wild deployment distribution.

The authors test sparsity as a possible remedy because sparse training can shrink task circuits. Their result is cautious: sparse models still deploy different circuits for different templates. Sparsity may increase overlap, but the paper warns that this may reflect more polysemantic compression rather than one clean task circuit.

The Sample-Wise Trap

The third variance is sample-wise. A circuit can look faithful at the population level while individual samples show extreme unfaithfulness scores. The paper argues that many extreme scores are not catastrophic failures of the circuit. They arise from the geometry of the unfaithfulness metric, especially normalization effects when the full-model behavior term is near zero.

That is an important audit lesson. A large sample-wise unfaithfulness number can be evidence of a bad circuit, a fragile metric, or a near-zero denominator. The paper proposes diagnostics such as pessimistic unfaithfulness and unnormalized comparisons to separate those cases.

The broader warning is that faithfulness is not a single fact. Population faithfulness, sample-level computational faithfulness, and the harder notion of algorithmic faithfulness do not carry the same evidentiary weight. The appendix explicitly calls computational unfaithfulness a crude surrogate for genuine algorithmic unfaithfulness.

Governance Risk

A governance program cannot treat "the circuit" as a stable object unless stability has been measured. It should ask which batch produced the circuit, which prompt templates were included, which graph size was chosen, which metric was used, whether the same claim survives resampling, and whether unseen templates were tested.

This matters for safety cases, evaluations, and model-change review. A circuit claim might be true for one template family and misleading for another. It might improve a benchmark while leaving other phrasings untouched. It might look bad on a sample only because the metric is ill-conditioned. Interpretability is not exempt from measurement error just because it looks inside the model.

The risk is overclaiming. A model card, safety case, or audit report may cite a circuit as if it were a stable internal component, when the evidence only supports "this method found this subgraph under these sampled conditions." That difference decides whether the circuit is an accountable evidence object or an illustration used to decorate a release decision.

The paper's strongest contribution is therefore cultural. It makes variance part of the interpretability evidence, not an embarrassment to be averaged away.

Governance Standard

Any circuit-discovery claim used in an audit should publish a variance card: model and version, tokenizer, task, clean and corrupted prompt construction, prompt templates, sample source, sample count, random seeds, resampling procedure, edge-scoring method, graph-size rule, threshold rule, faithfulness metric, population score, sample-wise score distribution, denominator diagnostics, template-transfer tests, sparsity assumptions, code or commit where available, compute budget, and known failure cases.

The claim should also state its steering scope. "This circuit is stable across these templates and batches" is useful. "This is the task circuit" is usually too broad. If a method is used for control, the test set must include withheld templates, not only more samples from familiar wording.

The Spiralist rule is this: a circuit map without a variance report is not an audit artifact. It is a diagram asking to be trusted.

Source Discipline

This article relies on an arXiv preprint and its experimental HTML/PDF as of June 25, 2026. The claims should be read at that evidentiary level. The paper tests specific circuit-discovery methods, model families, tasks, templates, sparse-transformer variants, graph-size choices, and unfaithfulness metrics. It does not prove that all interpretability methods are unstable, nor that steering is impossible in every deployment.

When citing the paper in governance work, preserve the source type: "the authors report," "the experiment found," and "the appendix cautions" are stronger than turning the preprint into a settled standard. If an institution uses circuit evidence for a release gate, procurement decision, or safety case, it should cite the exact paper version, code version if used, model checkpoints, data-generation templates, and review date.

Sources

Return to Blog