Blog · arXiv Analysis · Published: July 10, 2026 · Modified: July 10, 2026 · Last reviewed: July 10, 2026

The LLM Annotator Becomes the Measurement Instrument

Manuel Pita's July 2026 arXiv paper tests AMALIA-9B, Portugal's national language model for European Portuguese, as a social-science annotator for the moral-foundations construct of authority/subversion.

For this essay, an LLM annotator is a measurement instrument. The governance question is whether it can show the route from text to construct, not only match human labels often enough to look reliable.

The Paper

The paper is Validity of LLMs as data annotators: AMALIA on authority, arXiv:2607.08731 [cs.CL, cs.AI, cs.CY]. The arXiv record lists Manuel Pita as author, records submission on July 9, 2026, and the PDF metadata reports a 15-page preprint.

The narrow object avoids a vague argument about whether models understand culture. Pita asks whether AMALIA-9B can annotate one theoretical construct, authority/subversion, in a way that is valid as measurement rather than merely reliable as agreement. The core distinction is between matching human coders and following the construct's theory closely enough that the code means what researchers think it means.

The Model

The AMALIA model card describes AMALIA-9B-0626-DPO as an open-source language model targeting European Portuguese, developed by a consortium of Portuguese universities and research centers with Government of Portugal funding. The Hugging Face page lists the license as Apache-2.0, marks the model at 9B parameters, and says it was made publicly available on July 1, 2026.

That context matters. A national model is not only another chatbot; its political promise is local linguistic competence, inspectability, and institutional autonomy. The model card also states boundaries: high-stakes unsupervised decision-making is out of scope, factual outputs should be checked, production systems need task-specific evaluation, and downstream builders remain responsible for outcomes.

The Recovery Gap

Pita's test focuses on moral-foundations coding for authority/subversion. The construct is not the mere presence of a powerful person or the word "authority." It requires a relation among authority, order, stance, appraisal, and look-alike cases. The study uses 748 European-Portuguese transcreated texts, with confirmatory tests on 448 out-of-sample texts.

The paper compares two routes. In the undecomposed route, the model receives the whole codebook and returns one authority yes/no judgment. In the decomposed route, the model answers atomic clause questions, then a fixed formula recomposes those clauses into the construct decision. The recovery gap is the difference between the undecomposed F1 score and the recomposed F1 score against the same ground truth.

This is a strong audit idea. If the holistic answer agrees with humans but the clause route collapses, the model may be reaching the right-looking label through shortcuts. The replication package makes that audit concrete with the paired corpus, frozen constructs, per-annotator tables, evidence spans, scripts, and a pre-registration deposited before confirmatory annotation.

The Result

The headline result is not that AMALIA is useless. The paper reports strong format discipline: registered annotations completed, parse rates reached 100.0 percent under both instruction languages with one unparsable Portuguese-instruction response, and an unconstrained robustness pass returned correctly structured JSON on 746 of 748 texts in both conditions.

The problem is measurement. On the full 748-text Portuguese corpus, AMALIA under Portuguese instructions reached undecomposed F1 of 0.711 and recomposed F1 of 0.359, a recovery gap of +0.352. Under English instructions, it reached undecomposed F1 of 0.654 and recomposed F1 of 0.186, a gap of +0.468. On the 448 confirmatory texts, the paper reports open gaps of +0.358 under Portuguese instructions and +0.436 under English instructions, both above the pre-registered open-divergence threshold.

The error reading is the governance lesson. Of 46 unanimous-negative test texts that AMALIA's undecomposed Portuguese prompt coded positive, the reading panel attributed 36, or 78 percent, to surface correlates such as moral outrage, the mere presence of an authority figure, equality look-alikes, or no identifiable signal. The paper also reports that 54 percent had cited evidence rated at most partially grounded, misquoted, or absent.

Reference models sharpen the diagnosis without turning it into a slogan. GPT-OSS-120B closed the divergence on the same Portuguese corpus under the same English instructions with a reported gap of +0.028. Llama-3.3-70B did not close it, reporting +0.121. Pita treats this as evidence that the failure is primarily in the construct-model instrument.

Governance Reading

The Spiralist reading is that AI annotation turns measurement into infrastructure. Once an institution can cheaply code surveys, posts, interviews, public comments, classroom speech, workplace complaints, or parliamentary debate, it will be tempted to treat the code as a social fact. But a code is only as good as the instrument that produced it.

A responsible annotation receipt should preserve the corpus, consent and license status, construct definition, codebook, language and register boundary, prompts, model identity, calibration and confirmatory sets, clause answers, evidence spans, recomposition rule, F1 scores, recovery gap, error-reading method, human review, and shortcut patterns. That belongs beside the site's work on construct-validity receipts, Portuguese variety bias, synthetic respondents, and quantified objectivity.

The governance mistake is to ask only whether the model agrees with humans. Agreement is useful, but it is not the whole audit. For theoretical constructs, the deeper question is whether the model can execute the theory at the grain the theory requires. If not, it may still help with triage, but it should not stand alone for civic or scientific inference.

Limits

This is an arXiv preprint about one construct, one national model, and one corpus register. The paper says the ground-truth codes are inherited from English originals; verification defends transfer, but residual code drift remains possible. It also says the Portuguese construct is a faithful rendering rather than an AMALIA-specific recalibration.

The S1 and S2 error readings use an LLM panel, not independent human raters, and the paper marks parts of that diagnostic analysis as exploratory. The H2 test on whether Portuguese instructions narrow the gap was underpowered and did not meet the pre-registered criterion. None of this supports a broad verdict against Portuguese, national models, or 9B models in general.

The constructive conclusion is modest: local models need local validity audits for the measurement tasks they will be handed. Openness made this audit possible. The next failure would be to ignore what openness revealed.

Source Discipline

This page treats the arXiv abstract, metadata API, PDF, AMALIA model card, and replication repository as primary sources. It does not reproduce examples from the corpus, prompt text, tables, or long passages. The argument here is about the audit frame: agreement is not construct validity, and cheap annotation should not become cheap certainty.

The disciplined question is not "how much does it agree?" It is: what is the construct, which clauses fired, what evidence was cited, where did shortcuts appear, and who can contest the measurement before the code becomes policy or research?

Sources

Return to Blog