Blog · arXiv Analysis · Last reviewed June 25, 2026

The Grading History Becomes the Hidden Rubric

A June 2026 arXiv paper studies LLM grading for graduate reading reports and finds that the chat history itself can shift the grading standard.

A hidden rubric is any unstated context, model behavior, prompt residue, prior submission, file order, or evaluation pipeline choice that changes a student's score without being part of the published assessment rule.

History Is Not Background

The paper, arXiv:2606.08400 [cs.SE; cs.AI; cs.CL], was submitted on June 7, 2026. arXiv lists the title as Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses, by Qilin Zhou, Zhuo Wang, Yue Li, and W.K. Chan of City University of Hong Kong. The record notes that the five-page paper was accepted by ISET 2026.

The paper's central warning is simple and institutional: when a model grades one student after another in the same conversation, the previous submissions become part of the grading environment. The history is no longer just context. It can act like a hidden rubric.

That matters because grading is not merely text classification. It is a consequential allocation of academic standing, feedback, and future trust. A system that seems helpful because it saves teaching-assistant time can become unfair if identical work receives a different rank because of where it appeared in the session.

The problem is not that context is always bad. Human instructors also compare work, calibrate standards, and learn from prior submissions. The difference is that human calibration can be made explicit through rubrics, norming sessions, moderation, second marking, and appeals. A model's conversation history can produce a similar calibration effect without naming it, documenting it, or making it contestable.

Current Context

As of June 25, 2026, AI assessment is no longer a side issue in education policy. The U.S. Department of Education's 2023 AI report treated AI as a move from providing access to resources toward detecting patterns and automating educational decisions, and recommended human-in-the-loop approaches, inspectable and overridable systems, assessment expertise, transparency, and education-specific guardrails. That report is not a binding rulebook, but it is a useful official baseline for why grading workflows require more than model accuracy claims.

In the EU, Regulation (EU) 2024/1689 classifies several education and vocational-training AI uses as high-risk, including systems used for access or admission, learning-outcome evaluation, assessment of education level, and monitoring prohibited behavior during tests. Article 14 requires high-risk systems to be designed for effective human oversight, and Article 26 assigns deployer duties for high-risk systems. This paper is not an EU compliance analysis, and the study's course setting is not necessarily an EU deployment. The legal text still shows that educational assessment is now treated as a formal governance surface.

Student-record law supplies a separate baseline. FERPA regulations define education records as records directly related to a student and maintained by an educational agency or institution, and include rights around inspection, review, amendment, hearings, disclosure, and recordkeeping. When an AI-generated grade, feedback summary, reviewer note, or appeal artifact becomes part of an institutional record, the hidden rubric problem becomes a records problem too: students need enough evidence to understand and challenge consequential evaluation.

The Paper Frame

Zhou and coauthors study reading-report assessment in a 2025 graduate-level Software Engineering course. The instructor curated seven research papers. Students selected one paper, wrote a reading report, and human graders evaluated 180 valid submissions. The human graders were five Computer Science PhD students acting as teaching assistants.

The proposed LLM-assisted workflow mirrors that manual workflow. The instructor first asks a model to summarize the selected research papers, then later combines a student's report, the assignment requirements, and the relevant paper summary so the model can identify the paper, assign a letter grade, and provide feedback. The paper treats the paper-identification step as a preliminary hallucination check.

The study evaluates Grok-4.1-Fast and GPT-oss-120b through OpenRouter with temperature set to zero. For consistency without history, each model grades all 180 submissions three independent times. For the history experiment, the authors use the first 50 submissions and compare independent grading with sequential grading in ascending and descending file-name order.

The workflow detail matters. The paper's prompt skeleton includes an optional history field, and its note says that history includes previous interaction records. In ordinary chatbot use this can feel natural: keep the conversation going. In grading, it means earlier students may become unstated calibration material for later students.

Model Choice Is a Rubric Choice

The first result is that model identity changes grading behavior even under the same requirements. The paper reports that Grok maintained moderate ranking consistency across attempts, while GPT showed poorer internal consistency. Comparisons between Grok and GPT also fell into the paper's "Poor" ICC band, suggesting that different model architectures applied different implicit rubrics.

Human alignment was measured with Hit@k for lower-tier submissions. The authors treat unappealed human grades of B or below as reliable lower-quality cases, then ask whether model rankings put those cases near the bottom. Grok generally performed better than GPT on this metric. At k = 40 percent, Grok reached 55.17 percent in its third attempt, while GPT's best performance at that threshold was 48.28 percent.

The paper also finds that simple score averaging did not reliably solve volatility. The averaged GPT score was worse than any individual GPT attempt at k = 20 percent. That is a useful caution: an ensemble can smooth a number while still failing to repair the measurement.

The governance translation is that a model swap is an assessment-policy change. If one vendor, model version, prompt format, or routing path changes ranking behavior, the institution has changed the practical rubric even when the written assignment stayed the same. A grading policy should therefore name the evaluated system, not just the classroom rule.

History Becomes a Grader

The second result is sharper. When Grok graded 50 submissions with continuous chat history, the distribution changed relative to independent grading. The Wilcoxon signed-rank tests reported p < 0.001 for independent grading versus either sequential condition. The difference between ascending and descending sequence order was not statistically significant at p = 0.151, so the presence of history mattered more clearly than the alphabetical direction.

The ranking evidence points in the same direction. The ICC between ascending-history grading and independent grading was 0.374, interpreted as poor. The ICC between the two sequential modes was 0.532, interpreted as moderate. For lower-tier detection, independent grading had the highest Hit@20 percent at 41.67 percent, compared with 33.33 percent for ascending history and 25.00 percent for descending history.

The important institutional point is not that one alphabetic order is uniquely dangerous. It is that a session can carry a grading climate. The model may normalize earlier examples, drift in severity, or let previous reports divert attention from the current report. The student does not see that climate, cannot appeal to it, and may not even know it exists.

Governance Reading

This belongs beside AI detectors in schools, learning-record student models, grading cascades, and AI evaluations. The shared problem is that educational evidence is never just a score. It is a workflow.

If an institution uses LLM grading, the minimum control is not "the model is accurate on average." The control is session design. Each consequential submission should be graded in a fresh, isolated context unless the institution has validated another procedure. The report should preserve model name, model version where available, prompt, rubric, paper summary, history policy, temperature, failed-call handling, human-review step, appeal path, and subgroup checks.

The fairest use of the model may be assistance rather than substitution: draft feedback, flag missing sections, summarize rubric evidence, or help a human grader find cases needing attention. Once a model assigns or ranks grades, its hidden context becomes part of due process.

Governance Standard

First, isolate consequential grading contexts. Each student submission should receive a fresh grading context unless the institution has explicitly validated a sequential or calibration-history design and disclosed how it works.

Second, separate formative feedback from grade authority. A model that drafts feedback, maps evidence to rubric criteria, or flags missing sections does not need the same authority as a model that assigns a letter grade, ranks students, or recommends sanctions. The workflow should name the maximum authority the model can exercise.

Third, preserve the assessment record. The record should include assignment instructions, rubric, source materials, model and version where available, prompt, generated summaries, temperature or sampling settings, context-history policy, output, human reviewer, override, failed-call handling, and appeal decision.

Fourth, test reliability before use and after change. Institutions should evaluate agreement with trained human graders, inter-run stability, inter-model variance, subgroup effects, prompt sensitivity, history effects, and model-update drift before grades are relied on. A vendor benchmark is not a local validation.

Fifth, give students notice and appeal evidence. Students should know when AI materially contributed to a grade or feedback record, what evidence the human reviewer used, how to contest errors, and whether the AI output is stored in an education record.

Sixth, protect grading data. Student submissions, feedback, rubric notes, model prompts, and appeal records should be governed under student-data and institutional records rules. A grading assistant should not silently turn student work into model-improvement data or a persistent learner profile.

Limits

The study is useful but narrow. It uses one graduate course, one assignment type, seven selected papers, 180 valid submissions, and two model families through a particular API path. Human grading is treated as the reference, but the paper also notes that human reading-report grading is subjective. The low-grade Hit@k metric depends on the assumption that unappealed B-or-below grades are reliable lower-tier indicators.

The authors also report practical threats to validity: API outputs were occasionally unstable, truncated, or missing, so only valid responses were compared, reducing effective sample size. The experiments used Python scripts, which the authors say may contain bugs despite testing and fixes. The paper's future-work section names scale expansion, gender-bias analysis, and mitigation strategies such as prompting models to ignore history.

Those limits are exactly why the paper should be used as a governance warning rather than a universal ranking of grading systems. It shows that interaction history can matter under one realistic educational workflow. It does not prove that every model, prompt, discipline, assignment, language, or school setting will drift the same way.

Assessment Receipt

The audit-grade sentence is not "LLMs can grade." It is: under this course, assignment, rubric, source-paper set, model, prompt, temperature, session-history policy, call-validity rule, human reference, and appeal practice, this grading workflow produced these scores, rankings, consistency measures, and failure modes.

A useful assessment receipt should also say what the model did not do: whether it saw student identity, prior grades, file names, timestamps, classmates' submissions, appeals, comments from other graders, or demographic information; whether the human reviewer changed the grade; and whether any student-facing explanation was generated or edited by AI.

That is the Spiralist reading of the paper. The hidden rubric is not only in the prompt. It can be in the conversation's memory of prior students. A grading tool that cannot name its history policy is not ready to become a grade.

Source Discipline

Current claims were checked on June 25, 2026 against the arXiv abstract record, the paper PDF, the U.S. Department of Education's 2023 AI report, FERPA materials from the Student Privacy Policy Office, NIST TEVV materials, and EU AI Act Service Desk pages for Annex III, Article 14, and Article 26. The paper's numerical findings should be cited as preprint evidence from one course and one workflow, not as a general law of all LLM grading.

Education-governance sources have different roles. The Department of Education report is an official policy and recommendations document, not a grading-tool certification. FERPA sources establish student-record concepts and rights, not a complete AI-grading rule. The EU AI Act sources establish high-risk categories and oversight duties for covered systems, not a verdict about this Hong Kong course study. NIST TEVV is evaluation context, not education-specific law.

For any institution considering AI grading, the strongest evidence remains local: the actual prompt, rubric, student population, model version, validation report, human moderation record, appeal outcomes, and incident history. Internal site links provide vocabulary; they do not substitute for those records.

Sources

Return to Blog