Blog · arXiv Analysis · Last reviewed June 25, 2026

The Paper Assistant Becomes the Pre-Submission Referee

A June 2026 arXiv paper describes Google's Paper Assistant Tool as a review agent that finds technical flaws before submission. The useful boundary is not replacement. It is upstream evidence.

A pre-submission referee is an author-facing review system that inspects a manuscript before formal peer review while leaving authors, reviewers, editors, and program chairs responsible for the scientific claim and the publication decision.

The Referee Moves Upstream

Peer review is usually imagined as a gate at the end of writing. The paper on Google's Paper Assistant Tool, or PAT, moves part of that gate before submission. The tool reads a manuscript, produces a technical review, and gives authors a chance to fix errors before human referees see the paper.

That changes the social meaning of automated review. The question is not only whether an AI system can find a bug. It is whether the bug report becomes evidence, leverage, theater, hidden policy, or a new cost of appearing competent before a venue will even look at the work.

The clean definition matters. PAT, in the pilots described by the paper and official venue posts, is not a program-chair decision system. It is a private author tool placed before the formal review boundary. That makes it lower-stakes than an automated acceptance engine, but still consequential: it shapes the version of the manuscript that enters the record.

The Paper Frame

The source is Rajesh Jayaram, Drew Tyler, David Woodruff, Corinna Cortes, Yossi Matias, Vahab Mirrokni, and Vincent Cohen-Addad's Towards Automating Scientific Review with Google's Paper Assistant Tool, arXiv:2606.28277 [cs.LG], submitted June 26, 2026. The paper frames scientific validation as the bottleneck created when AI-assisted generation increases faster than human review capacity.

The authors do not present PAT as a conference decision-maker. In the pilots they describe, PAT is a pre-submission tool for authors. That distinction matters because an author-facing error finder has a different accountability shape than an automated acceptance system.

Current Context

The public record around PAT is broader than the arXiv paper. Google Research described the STOC 2026 experiment in December 2025 and later updated the post to name the system as Paper Assistant Tool. ICML announced a January 2026 experimental program in partnership with Google and said the feedback would be private to authors, separated from official review, stateless for model training, and subject to deletion and restricted access controls. ICML's March 2026 retrospective reported 869 author-survey responses and strong self-reported usefulness, including theory-gap and new-experiment signals.

NeurIPS announced an April 2026 PAT program for NeurIPS 2026 authors with the same basic boundary: optional author-side feedback before the deadline, no use in the official review process, and reviewers or program committee members unable to see the PAT feedback. NeurIPS also said its version incorporated parts of the ScholarPeer system, including stronger search, related-work comparison, and strengths/weaknesses analysis.

The governance context is not only "better paper feedback." ICMJE's AI-in-publishing recommendations say manuscripts are privileged communications and warn against uploading submitted manuscripts into AI systems where confidentiality cannot be assured without explicit author permission. ICML's 2026 LLM review policy separately distinguishes reviewer assistance from delegated judgment, requires reviewers to remain responsible for full review content, and restricts strength/weakness and review-writing delegation under its permissive policy. PAT sits on the author side of that boundary; moving the same system into reviewer or decision authority would require a different policy case.

What PAT Does

The paper reports that PAT ingests full manuscripts and focuses on objective checks: theoretical results, logical errors, experimental design, missing comparisons, and potential improvements. Its pipeline segments the paper into logical regions, allocates more compute to denser sections, runs specialized deep-review agents with the full paper in context, and uses a synthesis agent with search grounding before assembling the review.

The important product choice is orchestration. The authors contrast PAT with a single model call and with uncoordinated repeated calls. Their claim is that segmenting, budgeting, reviewing, and synthesizing can increase recall without forcing humans to sort through a flood of duplicated or hallucinated criticisms.

That architecture turns review into a workflow object. A good PAT-style output should not be read as "the model says no." It should be read as a set of claims about the paper: this proof step may fail, this experiment may need a missing comparison, this related-work claim may be too broad, this section may need clearer assumptions. Each claim needs a location, evidence, uncertainty, and a human decision about whether to revise.

Benchmark and Pilot Evidence

For benchmark evidence, the paper uses the SPOT benchmark's Math/CS equation-and-proof subset: 26 papers with 29 verified errors. Table 2 reports 21.1% detection accuracy for the original SPOT state of the art, 55.2% for a zero-shot Gemini 3.1 Pro run, and 89.7% for PAT with Gemini 3.1 Pro. The authors describe this as a 34% improvement over the zero-shot baseline, with human audit of the automated grader.

The comparison needs a caveat. The arXiv HTML says the authors used a logic-aware grader rather than the original SPOT paper's strict keyword-match grader, so the original SPOT number is context for difficulty rather than a perfectly apples-to-apples comparison. The narrow claim is still useful: on the filtered proof-error subset and under the authors' grading protocol, an orchestrated review pipeline found more verified errors than a single zero-shot model call.

For deployment evidence, the paper reports pilots with STOC 2026 and ICML 2026. Authors received one PAT review days to weeks before the final deadline, outside the formal peer-review process. Across the two programs, the authors report more than 4,700 reviewed submissions. The paper reports 97% of STOC respondents and 92.1% of ICML respondents would use PAT again; ICML's public retrospective separately reports 869 survey responses and the same 92.1% would-use-again figure.

The sharpest result is not the popularity metric. The paper reports that 11.6% of STOC respondents and 35.4% of ICML respondents said PAT identified substantive theory gaps. ICML's retrospective says 31% of authors with experimental results reported running new experiments because of the feedback. These are self-reported author outcomes, not independent proof that the final papers were more correct, more reproducible, or more likely to deserve acceptance.

The Automation Ladder

The paper's taxonomy is useful because it separates levels of authority. Role 1 is AI as a tool for authors, which is where the STOC and ICML pilots sit. Role 2 is AI as a tool for reviewers. Role 3 is AI as a supporting reviewer that produces an objective review for humans to assess. Role 3.5 adds ratings or recommendations. Role 4 is total automation of peer review.

This ladder prevents a common laundering move: citing a successful author-side tool as evidence for automated decisions. A pre-submission review that helps authors repair proofs is not the same thing as a model deciding whose work counts.

The taxonomy should be used as a governance boundary, not a maturity ladder. "Role 4" is not automatically the destination. Each step changes confidentiality, contestability, labor, appeal rights, and the distribution of scientific prestige. A tool that is legitimate as private author feedback can become illegitimate if the same output is made invisible to authors, used to rank papers, or treated as a substitute for expert disagreement.

Governance Reading

PAT points toward an evidence ledger for scientific work. A useful review agent should not merely output a verdict; it should preserve what claim was checked, what section was reviewed, what external source was used, what uncertainty remained, and what human chose to accept or reject the criticism.

The danger is authority drift. If authors come to treat the review as a required purification ritual, conferences may get cleaner manuscripts while also creating a new compute toll. If reviewers use similar tools without disclosure, rebuttal can become a fight against invisible machine criticism. If venues move to Role 3 or Role 4, the burden shifts from bug finding to institutional legitimacy: conflict policy, appeal paths, benchmark transparency, and access for researchers outside well-funded labs.

Several controls follow from the official posts and the wider peer-review rules. First, author-facing feedback should remain visibly outside the formal review process unless the venue explicitly changes policy. Second, manuscript text should move only through approved systems with disclosed retention, training, access, and deletion rules. Third, every AI-generated criticism should be contestable by the author who receives it. Fourth, any later reviewer-side or editor-side use should disclose the tool, input class, task, and decision point. Fifth, venues should not let AI-assisted polish obscure substantive weakness, because cleaner prose and fewer obvious proof holes can make a paper look more mature than its contribution warrants.

Equity is part of the safety case. If pre-submission review becomes expected, then access to compute, deadlines, OpenReview integration, queue priority, language support, and field-specific tuning become part of scientific opportunity. A free pilot can still establish a norm that later becomes expensive, uneven, or available first to researchers already near elite infrastructure.

Limits and Failure Modes

The paper names practical failures from pilot testing: date hallucinations and outdated knowledge, PDF parsing problems, and false claims that a proof or argument is wrong because the model misunderstood it. The authors say better search tooling and parsing addressed the first two categories, while reasoning failures remain a live limitation.

The benchmark is also narrow. A Math/CS proof-error subset is not all scientific review. It does not settle novelty, taste, social value, ethics, authorship, reproducibility, field-level disagreement, or whether a paper should be accepted. The strongest deployment claim is narrower: PAT can help surface technical defects before submission when humans remain responsible for the paper and the review process.

False-negative comfort. A clean PAT review can make authors believe the paper is technically sound when the tool missed a flaw outside its strengths.

False-positive repair work. A wrong critique can consume author time, push a paper into unnecessary detours, or cause authors to weaken a correct claim.

Polish masking. The obvious issues may be removed while deeper novelty, framing, ethics, or causal-inference problems remain harder for human reviewers to see under smoother presentation.

Confidentiality spillover. A private manuscript is still an unpublished knowledge object. Any tool path needs retention, access, training, deletion, and support-inspection boundaries.

Policy migration. A venue can begin with optional private author feedback and later treat similar outputs as reviewer aids, area-chair aids, triage filters, or ranking signals without rebuilding the legitimacy case.

Audit Receipt

The audit-grade sentence is: Jayaram, Tyler, Woodruff, Cortes, Matias, Mirrokni, and Cohen-Addad present PAT, report SPOT-subset and STOC/ICML pilot evidence, and propose a four-role taxonomy for AI in peer review; ICML, NeurIPS, Google Research, ICMJE, and ICML's LLM policy provide the surrounding institutional boundaries for privacy, disclosure, reviewer responsibility, and non-delegation of judgment.

A PAT receipt should record: manuscript version, submission venue, program window, author consent, PDF processing path, model or system version, review pipeline version, search or grounding tools, deletion schedule, access policy, generated comments, cited evidence, author actions taken, ignored comments, known failures, and whether the feedback was kept outside formal review.

The short receipt is: a pre-submission review agent can be useful scientific infrastructure only when evidence, limits, human authority, access conditions, confidentiality, and contestability stay visible.

Source Discipline

Use the arXiv paper for claims about PAT's architecture, SPOT-subset evaluation, STOC/ICML pilot results, taxonomy, and stated limitations. Use ICML, NeurIPS, and Google Research posts for program logistics, privacy boundaries, survey-retrospective claims, and official venue framing. Use ICMJE and ICML policy pages for confidentiality, reviewer responsibility, and non-delegation principles.

Do not collapse source types. A self-reported author survey is not an independent correctness audit. A benchmark on proof errors is not a full peer-review benchmark. A venue blog is not evidence that every implementation met the stated safeguards. A successful author-side pilot is not evidence for automated acceptance, automated rejection, or secret reviewer-side scoring.

Sources

Return to Blog