LLM evaluation reaches a governance desk in three forms when someone asks whether a language model is fit to deploy: a public benchmark score, the vendor’s evaluation report, and whatever your own team ran last week. All three get pasted into approval decks as if they answered the same question.
Each artefact was produced for a different purpose, carries different failure modes, and supports a different part of a governance file. Each section below grades one artefact against the same test: what can this artefact support in front of an auditor, and where does it break. The earlier version of this page, at LLM evaluation evidence before approval and release, listed artefacts to keep; this rewrite sorts them by what they are worth.
Artefact one: a public benchmark score describes a shared task set, not your deployment
Benchmark numbers are the cheapest LLM evaluation evidence in the stack and the most quoted. Their limits were measured last year: a NeurIPS 2025 review led by Andrew M. Bean of Oxford’s Reasoning with Machines Lab, with collaborators at Berkeley, Yale, Stanford and EPFL, examined 445 benchmarks from ICML, ICLR, NeurIPS, ACL, NAACL and EMNLP, coded by 29 experts against a 21-item questionnaire.
Three findings from that review decide how far an LLM evaluation score travels. Only 16.0% of the benchmarks “used uncertainty estimates or statistical tests to compare the results”, so most published comparisons cannot say whether a gap is real. Fewer than 10% used complete real-world tasks. And 7.7% drew their items solely from another benchmark, which compounds whatever the first one got wrong.
Contamination erodes what remains. The review’s own checklist tells benchmark authors to “maintain a held-out set of task items to facilitate ongoing, uncontaminated evaluation”, advice that exists because published items reach training corpora. Practitioners added a newer failure in September 2026 forum discussion: model providers have begun training agents to work inside particular runner scaffolds, so a score measured in a vendor’s scaffold reflects the pairing as much as the model.
A benchmark score supports one claim in a governance file: that a model family is plausible for a class of task. Deployment decisions, accuracy declarations and risk conclusions need more, and a governance pack resting on a benchmark score alone has no evidence at all.
Artefact two: a vendor evaluation report is a regulated disclosure with a defined minimum
Vendor documentation sits higher, because the AI Act specifies what an LLM evaluation disclosure must contain. Article 53(1)(a) requires providers of general-purpose AI models to keep technical documentation “including its training and testing process and the results of its evaluation”, meeting at minimum the elements of Annex XI, and Article 53(1)(b) requires a second pack for downstream providers containing at least the Annex XII elements.
Evaluation records for systemic-risk models carry more. Article 55(1)(a) requires those providers to “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks”.
Published government work shows the shape of an evaluation record that can be used. The Center for AI Standards and Innovation at NIST evaluated DeepSeek models against US reference models across 19 benchmarks in September 2025, reporting that agents built on one DeepSeek model were “on average, 12 times likelier than evaluated U.S. frontier models” to follow malicious injected instructions, and that the most secure DeepSeek model “complied with 94% of overtly malicious requests” against 8% for the reference models. Rates, comparators and a defined attack set are what make those numbers usable.
An LLM evaluation from the vendor supports supplier selection, contractual assurances and the limitations section of your own documentation. What it cannot do is describe your prompts, your retrieval layer, your thresholds or your users, which is the gap our foundation model vendor risk questions guide is built to close.
Artefact three: your own eval run is the only one that can carry a release decision
An eval run on your system is the artefact that regulators, auditors and litigators actually want, and the AI Act tells you when it has to happen. Article 9(8) requires testing “against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose”, carried out during development and “in any event, prior to their being placed on the market or put into service”.
Prior is the word in that eval run requirement that trips most teams. A threshold chosen after the score arrives proves nothing, so the approved metric, the pass mark, the approver and the approval date belong in the record before the run starts.
Eval run reproducibility is the second requirement, and it is not automatic. Thinking Machines Lab sampled 1,000 completions at temperature zero from one open-weight model on 10 September 2025 and reported “80 unique completions, with the most common of these occuring 78 times”, with divergence starting at token 103, because inference kernels are not batch-invariant and server load varies. An eval run therefore has to record provider, endpoint, model version, decoding parameters, batch and concurrency settings and date, or nobody can repeat it.
Statistics close the set. Item counts, confidence intervals and a paired test such as McNemar’s for pass or fail outcomes are what separate a regression from noise, a distinction a September 2026 forum thread made concretely when a commenter pointed out that a single task moving from 50 to 25 could be variance rather than decay.
The judge inside your eval run needs its own calibration table
Scoring by another language model is now common enough to need governing: the Oxford review found LLM-as-judge used at least partly in 17.1% of benchmarks, against exact matching in 81.3% and human ratings in 13.0%.
One published calibration table shows why an evaluation record needs disclosure. In a March 2026 arXiv preprint by Alexandre Cristovão Maiorano of Lumytics, 60 stratified cases were scored blind by two human evaluators and by an LLM judge from a different vendor than the system under test, with these pairwise results: humans agreed with each other 83.3% of the time at Cohen’s kappa 0.359; the judge agreed with the automated gate 63.3% at kappa 0.132; the judge agreed with the first human 80.0% at kappa 0.444 and with the second 73.3% at kappa 0.149; and one human agreed with the gate at kappa 0.000.
Read the eval run disagreements rather than the headline. Of 20 cases the gate rejected, 16 were latency violations invisible in response text, while the judge independently caught truncation, fabricated feature names, generic deflection and locale mismatches. The measuring instruments were looking at different things, which is an argument for keeping both and for recording which one produced each verdict.
Human ground truth inside an eval run is unstable too. Two trained evaluators at kappa 0.359 is fair agreement, not consensus, so a governance file that cites human review without an agreement statistic is citing an opinion. Judge model, version, scoring prompt, calibration sample and agreement figures belong beside every scored result.
What each artefact can support, side by side
| Artefact | Supports | Cannot support | Governing text |
| Public benchmark score | Shortlisting a model family for a task class | Deployment approval, accuracy claims, risk conclusions | None; 16.0% of 445 benchmarks report uncertainty |
| Vendor evaluation report | Vendor selection, contract terms, limitations in your documentation | Your prompts, retrieval, thresholds or users | AI Act Articles 53(1)(a), 53(1)(b), 55(1)(a) |
| Your own eval run | Release decisions, declared accuracy, post-market comparison | Claims about model families in general | AI Act Articles 9(6) to 9(8), 15(3), 72 |
| Judge calibration table | The weight a scored result carries | Replacing human review outright | Recorded practice; kappa 0.132 to 0.444 across pairs |
| Adversarial test results | Systemic risk and misuse conclusions | Ordinary capability claims | AI Act Article 55(1)(a) |
Article 15(3) turns your eval run into a published commitment
Accuracy figures stop being internal once a system ships. Article 15(3) states that “the levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use”, and Article 13(3)(b)(ii) requires those instructions to give the level of accuracy “against which the high-risk AI system has been tested and validated and which can be expected”.
Two consequences follow for the record itself. The declared number has to trace to a specific run on a specific version, because the instructions assert that the system was tested and validated at that level. And the population matters: a figure measured on curated test items and declared for general use is a claim the first incident will test.
Post-market monitoring then converts one run into a series. Article 9(2)(c) requires risk management to consider “the evaluation of other risks possibly arising, based on the analysis of data gathered from the post-market monitoring system referred to in Article 72”, so evaluations are re-run rather than filed once. Our LLM change management guide covers the triggers for re-running.
Release gates are where evaluation evidence becomes a decision record
An eval run that changes nothing is documentation, not governance. The Lumytics preprint reports a gate with three outcomes, promote, hold or roll back, across 38 evaluation runs and more than 20 internal releases in four weeks, with a question bank that grew from 59 to 133 scenarios.
Results show the gate earning its place. Thirty-six of 38 runs were promoted and two were rolled back when evidence coverage came in at 50% against an 80% threshold, and a Mann-Kendall test found task success falling over the period, with tau of minus 0.320 at p equal to 0.004, while latency rose with tau of 0.374 at p equal to 0.001.
Two cautions travel with those figures. The paper is a single-author preprint describing one internal system, so the numbers illustrate a method rather than establish a benchmark. And because the suite itself grew harder during the period, the declining trend line partly measures the suite, which is why the suite version belongs beside every chart.
Retention decides whether any of this is still evidence in 2034
Evaluation records have a shelf life set by law rather than by tooling. Article 18 keeps technical documentation for high-risk systems at the disposal of national authorities for ten years after the system is placed on the market, and Article 19 keeps automatically generated logs for at least six months.
Nobody else is guaranteed to hold the line. SR 26-2, issued by the OCC as Bulletin 2026-13 on 17 April 2026, states in footnote 3 that “Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance”, so bank model validation does not reach the assistant drafting credit memos, and the evaluation record becomes the main independent check that exists.
Govern365.ai holds the three LLM evaluation artefacts in their proper relationship rather than in one folder. The vendor pack attaches to the supplier record, the benchmark reference sits as context, and each eval run is stored against the model’s entry in the AI system inventory with its threshold, approver, judge calibration and release decision, so the version that shipped traces to the run that cleared it.
Retention is the part that pays for itself. Records are kept in export-ready form with dates intact across the ten-year clock, and the same file answers an ISO/IEC 42001 audit, a NIST AI Risk Management Framework mapping and a customer questionnaire. The wider artefact list sits in our AI audit evidence checklist.
Frequently Asked Questions
Can we rely on public benchmarks as governance evidence?
LLM evaluation by public benchmark is useful only for shortlisting. Fewer than 10% of 445 reviewed benchmarks use complete real-world tasks, 7.7% take their items from another benchmark, and 16.0% report uncertainty estimates or statistical tests.
Is a vendor’s evaluation report enough for deployment approval?
No. A vendor LLM evaluation is a regulated disclosure about the model under Articles 53 and 55, and it says nothing about your prompts, retrieval layer, thresholds or users.
What does the EU AI Act require from an eval run?
An eval run must satisfy Article 9(8), which requires testing against prior defined metrics and probabilistic thresholds before placing on the market, Article 15(3) requires declared accuracy levels in the instructions for use, and Article 55(1)(a) requires documented adversarial testing for systemic-risk models.
Why must the threshold be set before the run?
A pass mark chosen after seeing the score cannot support a claim about the system. The statutory wording is “prior defined metrics and probabilistic thresholds”.
Are LLM outputs reproducible at temperature zero?
Not by default. One experiment produced 80 unique completions from 1,000 samples at temperature zero, diverging at the 103rd token, because inference kernels are not batch-invariant.
How reliable is LLM-as-judge scoring?
LLM evaluation by judge model is variable, and the variability is measurable. In one published calibration, the judge agreed with one human at kappa 0.444 and another at kappa 0.149, while the two humans agreed with each other at kappa 0.359.
Does human review solve the reliability problem?
Not on its own. Fair agreement between two trained evaluators means a single reviewer’s verdict carries uncertainty, so agreement statistics belong in the record.
What adversarial evidence do regulators expect?
A defined attack corpus, results by category and a comparator. One 2025 government evaluation reported agent hijacking 12 times likelier for one model family and 94% compliance with overtly malicious requests against 8%.
How often should evaluations be repeated?
Re-run the LLM evaluation at every model, prompt or retrieval change, and on a cadence tied to post-market monitoring, because Article 9(2)(c) requires risk analysis to use monitoring data.
How long must evaluation records be kept?
Ten years of documentation availability under Article 18 for high-risk systems, and at least six months of automatically generated logs under Article 19.
Does bank model risk guidance cover generative AI evaluation?
No. SR 26-2, issued as OCC Bulletin 2026-13 in April 2026, places generative and agentic AI outside its scope.
What is the difference between a benchmark and an eval run?
LLM evaluation and benchmarking differ by ownership. A benchmark compares models on shared tasks. An eval run measures your system on your items against your threshold, and only the second supports a decision about your deployment.
