Blog · arXiv Analysis · Last reviewed August 12, 2026

The Name Prompt Becomes the Privacy Audit

A language model can attach a claim to a named person without showing whether the claim came from memorized text, a name-based inference, a retrieved source, or a population-level guess. A privacy self-audit can make that association reproducible enough to challenge; it cannot, by itself, prove what the model stores or which legal remedy applies.

The governance task is therefore larger than producing a transcript. An audit must identify the system layer, preserve the conditions of the test, protect the person from further disclosure, and route the evidence to correction, suppression, erasure, provider investigation, or appeal.

Category: AI privacy and auditing Published: June 25, 2026 Modified: August 12, 2026 Last reviewed: August 12, 2026 Tags: LLM privacy, self-auditing, GDPR, provenance, contestability

What an Association Audit Is

For this essay, a name-conditioned association audit is a repeated, time-stamped black-box test of whether an identified model or deployed system produces a candidate attribute more strongly or consistently for a particular name than it does across generic or counterfactual baselines. Its evidence is behavioral: prompts, outputs, comparison conditions, and stability across runs.

That definition keeps five different layers apart. An application may hold an explicit account memory. A retrieval system may fetch a webpage or customer record. A prompt may contain a fact supplied in the current session. A model may reproduce or infer an association from its parameters. A downstream product may then save the output as a profile field or use it in a decision. A single answer does not identify which layer caused it.

The distinction matters because a person can be harmed by a claim even when the claim is false and even when no verbatim training record can be shown. A guessed residence, native language, occupation, sexual orientation, religion, health condition, or criminal history can become a reputational, discrimination, safety, or access problem when a system routes it into search, ranking, eligibility, investigation, or agent tools. The immediate audit question is not “what does the model know?” It is: what claim did this system attach to this identity, under what conditions, and where did that claim travel?

The Paper

arXiv lists Dimitri Staufer, Kirsten Morehouse, David Hartmann, and Bettina Berendt’s Human-Centred LLM Privacy Audits: Findings and Frictions as arXiv:2603.12094v1 [cs.HC], submitted March 12, 2026. The paper identifies itself as accepted at the Human-centered Evaluation and Auditing of Language Models workshop at CHI 2026 and repeatedly describes its findings as interim results from an ongoing study.

The paper introduces LMP2, the Language Model Privacy Probe, as a browser-based self-audit tool. Its target is not an organization-wide privacy review and not a training-data extraction proof. It asks a narrower, user-facing question: what does a particular model associate with this name, and can the person receive evidence that supports interpretation, contestation, and a request for remedy?

How LMP2 Probes

LMP2 asks a user to enter a full name and choose attributes to test. The method adapts canary-style probing to black-box APIs by expressing a subject-property-value triple through up to five low-ambiguity paraphrases. It selects 50 properties from WikiMem’s larger set of 243 Wikidata properties, including date of birth, occupation, and phone number.

Because chat APIs generally score their own completions rather than arbitrary strings supplied by an auditor, the paper reformulates the test as fragmented sentence recovery. It truncates user-provided ground truths to two-character prefixes, generates 20 counterfactual prefixes, and asks the model to restore the final word or words. LMP2 aggregates frequency and, where available, probability evidence across those prompts, compares results with a generic-subject baseline, and reports association strength and confidence. In this tool, confidence means concentration of the observed outputs around a candidate value; it is not calibrated certainty that the candidate is true or memorized.

The method also creates privacy exposure. The paper says ground-truth values are kept client-side and not retained beyond the session, but the provider necessarily receives the names and prefixes submitted in the queries. Prefix truncation is an API adaptation, not anonymization. Any operational version therefore needs explicit consent, data minimization, a short retention schedule, restricted access to raw traces, and a rule against using a self-audit interface to investigate someone else.

Findings and Denominators

The paper reports a comparison across eight named models: Qwen3 4B Instruct, Llama 3.1 8B, Ministral 8B Instruct, GPT-4o, GPT-5, Gemini Flash 2.0, Grok-3, and Cohere Command A. The authors probe 100 famous public figures and 100 recombined, non-existent names. Confidence distributions separate the two sets across the evaluated models, but synthetic names still receive high-confidence defaults. The reported examples “ambidextrous” for handedness and “+1” for phone number show why a concentrated output is not the same as a person-specific fact.

The human work used adult EU residents recruited through Prolific: an initial survey of 155 people and two tool-based studies with a combined 303 participants from 19 EU countries. Sixty percent of the survey respondents expressed interest in a self-audit tool. In the tool studies, phone number and medical condition were each selected by fewer than 3 percent of participants, even though the survey identified those features among the leading concerns. The study therefore observes what participants felt safe enough to probe, not the full sensitivity surface.

The paper’s narrative says GPT-4o produced 11 of 50 features with at least 60 percent accuracy and reports an average of 45 percent across selected features. It lists sex or gender at 94.4 percent, sexual orientation at 82.9 percent, native language at 77.8 percent, eye colour at 74.3 percent, and hair colour at 74.1 percent. Those figures should not be read as population-wide capability rates. Feature selection was voluntary, denominators differed by attribute, and Appendix Table 2 shows that some of the 11 above-threshold rows rested on only 3, 8, or 10 selections.

There is also a version-one reporting inconsistency worth preserving in the record. The PDF narrative and Appendix Table 2 give slightly different percentages for sexual orientation, native language, eye colour, and hair colour. For example, the narrative gives 82.9 percent for sexual orientation and 74.3 percent for eye colour, while the table gives 83.61 and 72.22 percent. The direction of the result is similar, but exact subgroup percentages are provisional until the authors reconcile the table and prose.

Finally, 87 percent of outputs were not marked as privacy violations, while 72 percent of participants wanted an option to erase or correct model-generated information about them. This is an attitude result under the study interface, not a legal classification and not blanket consent. Avoidance of sensitive probes makes that distinction especially important.

Evidence, Not Provenance

The paper’s strongest contribution is not its accuracy table. It is the warning that output-based audits establish an elicited association, not provenance. A correct completion might reflect memorized text, an inference from the name, indirect identifiers, a retrieved source, prompt leakage, or a population-level prior. An incorrect completion can still be stable enough to cause harm. Neither result reveals the causal path from output alone.

The paper identifies nine linked frictions: ambiguity about audit scope; participant selection effects; entanglement of memorization, inference, and guessing; name ambiguity and indirect identification; multiple or time-varying ground truths; attributes that are subjective, relational, or evaluative rather than discrete facts; English and Latin-script limits; deployment architectures that mix model, retrieval, and tools; and the difficulty of turning probabilistic evidence into action. These are not footnotes to remove with a larger sample. They define what the audit can responsibly certify.

A useful result statement is therefore bounded: “Under these prompts, baselines, model endpoint, settings, and date, this candidate appeared with this measured stability.” It is not: “the model remembers this person,” “the training set contains this record,” or “the system knows this fact.” Stronger claims require complementary evidence such as training-data records, retrieval logs, system traces, source documents, data-flow maps, or provider investigation.

Current Governance Context

The legal context does not turn every suspicious completion into an automatic deletion order. Under the GDPR, access, rectification, erasure, restriction, and objection have different triggers, limits, and responsible controllers. A prompt transcript can help a person identify an issue, but it does not by itself establish which processing operation contains personal data, whether the relevant model is anonymous, who controls each layer, or which exception or remedy applies.

The European Data Protection Board’s Opinion 28/2024 rejects a blanket assumption that models trained on personal data are anonymous. It says anonymity must be assessed case by case and requires both the likelihood of extracting personal data about training subjects and the likelihood of obtaining such data through queries to be insignificant, considering means reasonably likely to be used. An LMP2-style trace may be relevant evidence for that assessment, but it is not the assessment by itself.

France’s data protection authority, the CNIL, made the operational connection explicit in its January 2026 guidance on data-subject rights and AI. It says rights must be supported in training datasets and in models that are not anonymous, recommends a documented internal query process to assess what data a generative model may hold about a person, and distinguishes dataset correction from its delayed or absent effect on an already trained model. The CNIL describes retraining, robust system-level output measures, and clear communication to the requester as possible parts of a proportionate response. The UK Information Commissioner’s Office separately warns that output filters may be useful but do not themselves remove data from a model.

For risk management rather than legal entitlement, NIST’s voluntary Generative AI Profile recommends periodic monitoring of generated content for personal or sensitive-data exposure, documented feedback routes for negative impacts, and avoidance of capability claims extrapolated from narrow or anecdotal tests. That is the right level of claim for this paper: a self-audit can be one monitoring instrument inside a larger governance system.

Governance and Safety Standard

Keep the audit subject-controlled. A self-audit interface should verify that the requester is testing their own identity or has a legitimate representative role. It should not become a cheap dossier tool for employers, landlords, insurers, schools, abusive partners, political campaigns, or data brokers. Sensitive categories should use stronger consent, identity, access, rate-limit, and deletion controls.

Quarantine unverified associations. A name-conditioned output should never become a profile fact merely because it is confident or repeatable. It should not enter hiring, credit, housing, insurance, health, education, policing, moderation, safety, or access decisions without an independent, authorized source and a meaningful review path. When such an association has already influenced a consequential action, the response must include notice, suspension of further use, human review, and repair rather than only a model-side patch.

Route the remedy to the layer that failed. An explicit memory or database error calls for correction and propagation checks. A retrieval error calls for source correction, index refresh, or source exclusion. Suspected model-level retention calls for provider investigation and may support retraining, unlearning, or robust output mitigation, depending on the evidence and applicable law. A downstream decision calls for notice and appeal. Suppression, deletion, rectification, and machine unlearning are not synonyms, so the response must say what changed and what remains.

Make repeatability visible. The evidence package should preserve prompts, paraphrases, generic and synthetic baselines, model and endpoint identifiers, retrieval state, tool availability, settings, dates, call counts, and failed or refused runs. It should distinguish the user’s ground truth from the model’s candidate and show instability rather than averaging it away. The goal is a contestable record, not an aura of certainty.

Minimize the audit’s own privacy footprint. Raw prompts can contain names, sensitive attributes, counterfactuals, and the user’s corrections. Store only what the remedy requires; separate identity from research metrics; encrypt restricted evidence; expire raw traces; prohibit training reuse without separate authority; and record which provider received each query. An audit that leaks the very attributes it is meant to investigate has failed at the first boundary.

Limits

This is a version-one workshop paper reporting interim findings from an ongoing study. Its user evidence comes from adult EU residents recruited on Prolific, and the tool studies used self-selected attributes. The evaluation does not represent children, non-EU legal contexts, people who declined sensitive probes, or the many users whose names and languages do not fit English and Latin-script assumptions.

The model results are endpoint- and time-specific. They do not establish current behavior for later product versions, different system prompts, safety layers, sampling settings, retrieval sources, or agent tools. The famous-versus-synthetic comparison validates a useful separation signal under the authors’ method; it does not validate every candidate value as a person-specific memory.

Those limits do not make the exercise useless. They locate its proper role. A prompt transcript is an initial evidence artifact that needs metadata, comparison conditions, uncertainty, user interpretation, system-layer diagnosis, provider response, and a repair channel before it becomes an accountable privacy process.

Privacy Audit Receipt

A privacy audit receipt should record: requester authority and consent; selected attribute; subject identifier and any disambiguating context; prompts, paraphrases, prefixes, and counterfactuals; generic and synthetic baselines; application, provider, model, endpoint, and version; system prompt, settings, retrieval and tool state; timestamp and call count; refusals and errors; top candidates; association strength and confidence as defined by the method; run-to-run stability; user-supplied truth and dispute status; observed downstream use; systems and recipients searched; and the requested remedy.

The disposition should use precise verbs: corrected in source data, removed from retrieval, suppressed at output, erased from a named store, queued for retraining, escalated for provider investigation, reversed in a downstream decision, rejected with reasons, or no action. It should also state whether the remedy propagated to recipients and whether retesting under the same audit specification still reproduces the association.

Source Discipline

The paper is the primary source for LMP2’s method, model list, samples, and reported findings. Its results are not a census of model behavior, a provider disclosure about training data, or a legal determination. This review checked the arXiv abstract, version-one PDF, and experimental HTML on August 12, 2026; arXiv listed only v1. Where the narrative and Appendix Table 2 disagree, this essay reports the discrepancy rather than silently choosing a percentage.

The GDPR text establishes legal rights. EDPB and CNIL documents provide regulatory interpretation and operational guidance; they do not convert every model output into proof that a particular record exists in model weights. NIST AI 600-1 is voluntary risk-management guidance, not law. The ICO source is used only for its distinction between output filtering and removal from a model.

Future reviews should recheck the paper version, model endpoints, regulator guidance, and linked remedies. Product names in the study identify the systems the authors tested; they should not be read as claims about those products’ behavior after the study.

Sources

Return to Blog