Blog · arXiv Analysis · Last reviewed June 25, 2026

The LLM Label Becomes the Review Tax

Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Julian Frattini, and Philipp Leitner's June 2026 arXiv paper on LLM-labelled code review shows that provenance labels are not neutral stickers. They change attention, create verification work, and turn the prompt into a missing review artifact.

Here, the review tax is the extra human verification burden created when a code label signals AI assistance but does not provide segment boundaries, generation context, prompt traceability, required checks, author modifications, and accountable merge authority.

The Label Is Not a Warning Label

The paper, arXiv:2606.26505v1, was submitted on June 25, 2026. arXiv lists the exact title as Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code, by Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Julian Frattini, and Philipp Leitner, with categories cs.SE and cs.HC. The arXiv comment says it was accepted at the 41st IEEE/ACM International Conference on Automated Software Engineering, ASE 2026.

The governance question is narrower than whether developers should use code generators. It asks what happens when a reviewer is told that a code segment came from an LLM. Many organizations want provenance labels because hiding machine assistance is risky. But the label is not a magic warning. It changes the reviewer's attention and creates a new kind of work: deciding how to inspect code whose origin carries a special suspicion.

That makes the label part of the review interface. A provenance mark can support accountability, but it can also become a vague anxiety trigger if the organization has not defined what extra review means. The label says "look here." It does not say what evidence would make the code acceptable.

A useful label is therefore not just a disclosure bit. It is a claim about scope, origin, and responsibility: which lines or files were generated or assisted, what task or prompt shaped them, what the human author changed, what tests and checks ran, what risks remain, and which reviewer or maintainer has authority to accept the residual risk.

Current Context

As of June 25, 2026, this question matters because AI coding has moved beyond autocomplete. GitHub's current Copilot cloud-agent documentation says the agent can research a repository, create an implementation plan, make code changes on a branch, and optionally open a pull request; it also describes usage metrics for pull requests created by the cloud agent. That product surface does not prove that agent-generated pull requests are good or bad. It shows why provenance labels now sit inside ordinary repository governance, not only inside an editor.

Secure-development governance already has a vocabulary for this. NIST's Secure Software Development Framework describes risk-based practices for producing well-secured software, protecting software from tampering, tracking requirements and design decisions, collecting provenance data for components, and responding to vulnerabilities. NIST's SP 800-218A community profile adds generative-AI and dual-use foundation-model considerations across the development life cycle. For LLM-labelled code, the governance lesson is direct: the label should connect to secure-development evidence, not substitute for it.

The current pressure is not simply that reviewers mistrust machine output. It is that review queues can now receive generated code, generated tests, generated documentation, generated review comments, and agent-prepared pull requests. A one-word "AI" label can make every reviewer improvise their own safety ritual. That is the tax.

What the Experiment Measured

The authors conducted a Wizard-of-Oz experiment with 32 participants. Participants reviewed four Python files presented as pull requests during one-hour sessions while an eye tracker recorded gaze behavior. The files averaged 77 plus or minus 10 lines. Across the tasks, selected areas of interest appeared either with an LLM-generated label or without one, and label assignments were rotated so the same code could be compared across participants under labelled and unlabelled conditions.

The label was designed to feel realistic. It included a comment header indicating that the code was generated by an LLM, along with metadata such as generation time and the prompt used to produce the code. The authors then used Bayesian data analysis on fixation durations and saccade lengths, and combined that with qualitative analysis of gaze paths and exit interviews.

This design matters because it separates perceived provenance from code quality. The study is not simply comparing human-written code to machine-written code. It asks how reviewers behave when the same review task contains an origin signal that changes what the code means institutionally.

It also keeps the claim bounded. The task was a controlled review exercise, not a production merge decision with CI, ownership files, security scanners, incident history, release pressure, and team norms attached. The result is strongest as evidence about reviewer attention under a provenance signal, not as a universal measurement of all code review.

Same Scrutiny, More Time

The headline finding is precise. The label did not produce a practical change in review thoroughness as measured by saccade length. Participants still scanned labelled and unlabelled code with broadly comparable thoroughness. But they spent more time fixating on code labelled as LLM-generated, with the paper reporting fixation-time increases up to 60 percent for more complex code in the study setup. The paper reports a credible effect for the LLM label on fixation duration, while file order and carryover effects were not credible in the same analysis.

The interviews sharpen that result. Twenty participants, or 62.5 percent, reported no change in their evaluation criteria between labelled and unlabelled code. Twelve participants did adjust criteria, often looking differently at logical correctness or code quality depending on their assumptions about LLM mistakes. Fourteen participants used the prompt during review.

The Spiralist lesson is that human oversight is not just a role assignment. It is a cognitive workload. A reviewer can sincerely believe they are applying the same standard while their eyes reveal a different cost profile. The LLM label becomes the review tax: extra attention without necessarily a clearer rule for acceptance.

The Prompt Becomes an Artifact

The prompt was not merely background information. Some participants treated it like a requirement: they compared the generated code against what the prompt asked for. Others treated it more like documentation, using it to understand the intent behind a code segment. In both cases, the prompt became part of the review material.

This is one of the paper's strongest governance contributions. If a team labels code as LLM-generated but withholds the prompt, the reviewer sees risk without seeing the instruction that shaped the output. If the prompt is embedded as a code comment, it can clutter the file and create maintenance problems. The authors argue instead for prompt-to-code traceability as metadata that can be accessed on demand, especially for multi-turn generation where one prompt is not the whole history.

Traceability does not mean pasting every private chat message into source control. Prompts may contain secrets, proprietary context, customer data, or irrelevant developer notes. The governed artifact can be a redacted prompt, a task summary, a conversation reference, or an access-controlled trace. The key is that a reviewer with legitimate authority can reconstruct intent and generation context when the label makes that context relevant.

Policy Boundary

A serious AI coding policy should therefore define what LLM-labelled code requires from reviewers. Does the reviewer check prompt alignment, security properties, tests, edge cases, licensing, performance, maintainability, or all of them? Who owns that check when the original author used an assistant? What must be preserved when the generated code is modified? When does a label expire because the human has rewritten enough?

The boundary should also distinguish assistance modes. Inline completion, generated helper function, generated test suite, generated migration, agent-created branch, and autonomous pull-request preparation impose different review duties. A label that collapses all of them into "AI-generated" is too blunt for policy and too vague for maintainers.

This connects to Codex workflow reorganization, machine contributor maintainer tax, the contributor ladder and agent queue, human oversight of AI systems, and AI audit trails. A code provenance label is only useful when it is tied to authority, evidence, and action.

Governance Standard

First, label by segment and mode. The label should identify the affected lines, files, generated assets, tests, or documentation, and it should distinguish assisted human work from generated human-submitted work and agent-prepared pull requests.

Second, attach traceability without polluting the code. The prompt, task, model or tool, generation time, conversation history where relevant, and human edits should be available through review metadata or an audit trail, with secrets and irrelevant personal data redacted.

Third, bind the label to verification criteria. The review checklist should say whether the reviewer must check prompt alignment, logic, edge cases, security properties, dependency changes, licensing, performance, accessibility, tests, documentation, and rollback. "Be extra careful" is not a policy.

Fourth, keep responsibility with an accountable human and a defined merge authority. The author or sponsor should be able to explain, test, repair, and stand behind the code. The reviewer should not inherit responsibility for reconstructing a hidden generation process.

Fifth, measure the tax. Teams should track reviewer time, repeated review rounds, CI minutes, defects found in labelled code, reverts, incidents, security findings, and maintainer opt-outs. If the label raises suspicion without supplying evidence, it shifts cost onto the reviewer rather than improving safety.

Limits

The paper is a controlled study, not a census of all software review. The authors note external-validity risks: participants' stance toward AI may vary, eye-tracking equipment may affect behavior, and the lab task cannot fully reproduce industrial review settings. The labels were also intentionally constructed for the experiment, so real company tooling may produce different effects.

Those limits do not weaken the practical point. If provenance labels change attention under controlled conditions, teams should not assume labels are neutral in production. The right response is not to hide AI use. It is to test how labels interact with review tools, policy, team norms, and maintainer workload.

The result should also not be converted into a universal coefficient. The study supports a governance question, not a fixed surcharge. Different languages, team trust, model tools, review interfaces, domain risks, and repository policies may change whether the label saves time, wastes time, or moves attention to the right place.

Review Receipt

A review receipt for LLM-assisted code should name the generated segment, model or tool if known, generation context, prompt or conversation trace, author modifications, tests run, reviewer role, required verification criteria, reviewer comments, unresolved concerns, and final merge authority. The prompt should travel as review metadata, not as folklore in a chat window.

The receipt should also record what was not checked. If the reviewer did not examine licensing, dependency risk, security properties, generated tests, performance, localization, or accessibility, the record should say so rather than implying total assurance. A receipt with honest gaps is better governance than a label that sounds complete.

The label becomes the review tax when it asks for suspicion without supplying a method. The better artifact is a label plus a trace: this code was assisted, this is what was asked, this is what changed, this is what was checked, this is what remains uncertain, and this is who accepted the remaining risk.

Source Discipline

Current claims were checked on June 25, 2026 against the arXiv abstract page, arXiv experimental HTML, and paper PDF. The article treats the paper as a current arXiv preprint with an ASE 2026 acceptance note, not as a universal production standard. Its participant count, review-task design, eye-tracking results, prompt-use findings, and validity threats are paper claims under one controlled design.

GitHub documentation is used only to establish that repository-level coding agents can work on branches and optionally create pull requests inside a live platform workflow. It is not used as evidence that such agents reduce net maintainer workload. NIST SSDF materials are secure-development context, not proof that an LLM label alone satisfies secure-development governance.

Internal links provide site vocabulary for maintainers, audit trails, coding agents, and human oversight. They are not substitutes for the paper, product documentation, or standards material listed below.

Sources

Return to Blog