Gartner predicts that 40 percent of organizations deploying AI will implement dedicated AI observability tools by 2028 to monitor model performance, bias, and outputs, according to a May 2026 Gartner press release. That figure matters because observability only tells you a model is drifting. It says nothing about whether the model should have shipped in the first place.
Every large language model release now has an audience that was not there two years ago: a customer security questionnaire, an internal audit committee, or a state regulator asking what testing happened before go-live. The question is rarely whether you tested the model. It is whether you can produce the evidence.
This piece covers what evaluation evidence actually needs to contain, how ISO/IEC 42001, the EU AI Act, and NIST AI RMF each define it, and how to build a release gate that produces it automatically instead of scrambling for it after the fact.
What Counts as Evaluation Evidence, and What Doesn’t
A benchmark score in a Slack message is not evidence. Evidence is a record that survives the person who created it leaving the company: a documented test plan, the metrics used, the acceptance threshold, the actual results, and a named person who reviewed them.
Most teams already run some form of evaluation before an LLM ships. Where the process breaks down is documentation. Engineering runs a batch of prompts through an eval harness, checks that the pass rate looks reasonable, and moves on. Six months later, an auditor asks for the pre-release evaluation record for that model version, and nothing exists that ties the test run to a specific version, a specific threshold, or a specific approval.
Treat evaluation evidence as having four required components: what was tested (task, dataset, model version), how it was tested (method and metric), against what standard (a documented pass or fail threshold), and who signed off. Remove any one of the four and the record stops functioning as evidence in an audit or procurement context.
NIST AI RMF’s MEASURE Function: The Evidence Backbone
NIST AI RMF 1.0 organizes AI governance into four functions: GOVERN, MAP, MEASURE, and MANAGE. MEASURE is the function that produces evaluation evidence directly, and it is also the one organizations most often implement on paper without operationalizing. The framework treats GOVERN as organization-wide and applies MAP, MEASURE, and MANAGE at the level of the individual AI system.
For generative AI and LLM deployments specifically, NIST published AI 600-1, the Generative AI Profile, in July 2024. It adds twelve risk categories that sit inside or extend the base MEASURE function, including confabulation (the formal NIST term for hallucination), data privacy leakage, and prompt injection. If your evaluation package does not address these twelve categories, it is not a complete MEASURE evidence set for an LLM.
A compiled evidence package for MEASURE typically includes pre-deployment evaluation reports, ongoing monitoring data once the system is live, and a documented methodology explaining the scoring criteria and reassessment frequency. The most common gap is a risk register that lists risks as medium with no data behind the rating. That is an assessment without measurement, and it does not hold up when MANAGE later has to justify a risk treatment decision.
ISO/IEC 42001: Clause 8.4 and Clause 9.1
ISO/IEC 42001:2023 splits the evaluation question across two clauses. Clause 8.4, the AI system impact assessment, asks a broader question than a technical benchmark: what happens to people, the organization, and society if this system behaves as tested, and what happens if it does not. Clause 9.1, monitoring, measurement, analysis and evaluation, is where the technical performance data lives.
An auditor working through Clause 9.1 wants to see thresholds and metrics defined before the model went live, not selected afterward to match whatever the results turned out to be. Clause 9.1 works alongside Clause 8.2, the operational AI risk assessment, and both clauses draw on the Annex A control set for data quality, system impact, and third-party relationships, which an organization implements and then references directly in its evaluation records.
The practical failure mode here mirrors the NIST gap: organizations complete an impact assessment once, at initial certification, and never revisit it. Clause 8.4 explicitly expects reassessment after significant changes, which for an LLM means every model version upgrade, every material prompt or system-instruction change, and any expansion into a new use case.
EU AI Act Article 15: Accuracy, Robustness and Cybersecurity
Article 15 of Regulation (EU) 2024/1689 requires high-risk AI systems to reach an appropriate level of accuracy, robustness, and cybersecurity, and to perform consistently across their lifecycle, not just at the point of initial testing. This is not a general aspiration. Providers must declare accuracy metrics and test datasets in the instructions of use, which means the acceptance threshold has to exist in writing before deployment, not as an informal team consensus.
Robustness testing under Article 15 goes beyond standard QA. It expects stress testing, out-of-distribution testing, and, where the system takes user-controlled input, adversarial red-teaming. For an LLM, that last category covers prompt injection and jailbreak resistance directly, and it should be read alongside Article 9, which requires this testing to sit inside a documented risk management system rather than as a standalone QA exercise.
Article 15 applies at the design and pre-market stage. Article 72 picks up where Article 15 leaves off, requiring a post-market monitoring plan that verifies accuracy and robustness are maintained after deployment. The two articles form a single lifecycle obligation: prove it works before release, then prove it keeps working.
A frequent point of confusion: a general-purpose LLM is not automatically a high-risk AI system. General-purpose models carry their own transparency obligations, and systemic-risk models face additional evaluation duties under separate articles. An LLM inherits Article 15’s high-risk testing requirements when it is deployed inside a specific high-risk use case listed in Annex III, such as employment screening, credit decisions, or law enforcement risk scoring. The obligation attaches to the deployment context, not the model itself.
The Cross-Framework Evaluation Evidence Map
Most compliance teams working across all three frameworks end up building three separate evidence trails because nobody has mapped the overlap. That duplicates work and, worse, creates three slightly different versions of what accuracy testing means for the same model. The table below aligns the evaluation requirements that recur across ISO 42001, the EU AI Act, and NIST AI RMF, so a single test run can satisfy all three where the underlying control is the same.
Cross-framework mapping: what each framework requires as LLM evaluation evidence before release
| Evaluation requirement | ISO/IEC 42001:2023 | EU AI Act (Reg. 2024/1689) | NIST AI RMF 1.0 |
|---|---|---|---|
| Documented pre-release testing plan | Clause 8.1 operational planning and control | Article 9(6)-(7) testing to identify and address risks | MEASURE 1.1: approaches and metrics identified and documented |
| Accuracy and performance metrics | Clause 9.1 monitoring, measurement, analysis and evaluation | Article 15(1)-(2) accuracy declared and measured against benchmarks | MEASURE 2.5: AI system performance validated pre-deployment |
| Bias and fairness testing | Annex A data quality and impact-assessment controls; Clause 8.4 | Article 10 data governance; Article 15 performance consistency | MEASURE 2.11: fairness and bias evaluated |
| Adversarial and security testing | Clause 8.2 AI risk assessment; Clause 9.1 | Article 15(4)-(5) resilience to third-party manipulation | MEASURE 2.7: AI system security and resilience evaluated; NIST AI 600-1 risk category 8 |
| System-level impact assessment | Clause 8.4 AI system impact assessment | Article 9(2) risk management system covering foreseeable misuse | MAP 1.1, MAP 5.1: context and impacts characterized |
| Independent review before go-live | Clause 9.2 internal audit; Clause 5 leadership sign-off | Article 43 conformity assessment (high-risk systems) | GOVERN 4.1: organizational accountability for AI risk decisions |
| Ongoing monitoring after release | Clause 9.1 continued monitoring; Clause 10 improvement | Article 72 post-market monitoring plan | MANAGE 4.1: post-deployment monitoring and incident response |
Read across a row, not down a column. A single adversarial red-team exercise, run once and documented properly, produces evidence for ISO Clause 9.1, EU AI Act Article 15(4)-(5), and NIST’s MEASURE 2.7 simultaneously. The work is identical. Only the label and the filing location change.
For systems using retrieval-augmented generation, evaluation should also cover RAG governance controls across data sources, retrieval, and generated outputs.
Building the Pre-Release Evaluation Package
A complete pre-release package for an LLM covers five test categories, and skipping any one of them is the most common reason evaluation evidence gets rejected in an audit or a customer security review.
Accuracy and grounding. Measure factual correctness against a task-representative dataset, not a generic public benchmark. A model that scores well on a standard leaderboard can still fail badly on your specific document types or domain terminology.
Hallucination and factuality. Test the rate at which the model fabricates citations, numbers, or claims that are not supported by the input it was given. NIST AI 600-1 treats this as its own risk category rather than folding it into general accuracy, and your evidence should do the same.
Bias and fairness. Test outputs across the demographic or use-case segments relevant to your deployment. A hiring-adjacent LLM feature needs disparate-impact testing even if the model itself was never trained specifically for hiring.
Adversarial robustness and security. Red-team for prompt injection, jailbreak attempts, and data exfiltration through crafted inputs. This is the category most often skipped because it requires adversarial thinking rather than standard QA, and it is exactly the category Article 15(4)-(5) and NIST MEASURE 2.7 both single out.
System-level impact. Go beyond the model and assess the deployment: who is affected if the model is wrong, what the fallback process is, and whether a human reviews outputs before they reach a consequential decision. This is the Clause 8.4 and Article 9(2) layer that a purely technical evaluation misses.
Where Evaluation Evidence Falls Apart in Practice
The gap is rarely a lack of testing. It is a lack of a durable record. Three patterns show up repeatedly across audits and vendor security reviews.
The vanishing threshold. A team defines a pass rate before testing, the model falls short, and the threshold quietly moves to match the result rather than triggering a documented exception. An auditor comparing the original test plan to the final sign-off catches this immediately, and it is one of the fastest ways to turn a minor finding into a credibility problem.
Testing the wrong version. Evaluation happens against a staging model, and the production deployment quietly points at a newer model version by the time it ships, often because a vendor pushed an update. The evidence exists, but it does not describe the system actually in production.
No independent sign-off. The engineer who built the fine-tuning pipeline is also the person who approves the evaluation results. Both ISO 42001’s Clause 5 leadership requirements and NIST’s GOVERN function expect accountability that sits outside the build team. This is one of the most frequently cited findings in early ISO 42001 audits [VERIFY: confirm current-year audit finding frequency data before publication], and it is also the easiest gap to close, since it requires a process change rather than new tooling.
Evaluation should not stop at initial approval. A defined LLM change management process helps determine when a new model version or system modification requires another evaluation.
Operationalizing the Release Gate
A release gate turns evaluation from a one-time exercise into a repeatable control. The mechanics matter less than the discipline: no model version reaches production without the evidence package attached to that specific version, reviewed by someone independent of the build, before deployment is authorized.
- Define the evaluation requirements and thresholds for a model class before development starts, not after testing produces a result.
- Run the five test categories against the specific model version and configuration going to production, including any retrieval or tool integrations.
- Document results against the pre-defined thresholds, with any exception explicitly approved and recorded rather than absorbed silently.
- Route the package to an independent reviewer for sign-off, distinct from the team that built or fine-tuned the model.
- Attach the evidence package to the release record so it is retrievable by model version, not just by project or team.
- Set a re-evaluation trigger for version changes, material prompt changes, and a fixed calendar interval regardless of whether a change occurred.
This is where most manual processes lose the thread: evidence collected in scattered documents, spreadsheets, and chat threads becomes nearly impossible to retrieve six months later when a customer, auditor, or regulator asks for the record tied to a specific model version.
Govern365.ai‘s AI model registry attaches evaluation evidence directly to each model version in the inventory, and its audit evidence management module keeps the test plan, results, and sign-off retrievable by version rather than buried in a shared drive. The compliance dashboard then flags any production model whose evaluation record has aged past its re-assessment window, which is the failure mode that most often turns into an audit finding.
Pre-release LLM evaluation evidence checklist
- Documented test plan naming the model version, intended use, and evaluation owner (NIST MEASURE 1.1 / ISO Clause 8.1)
- Accuracy and grounding results against a defined benchmark and acceptance threshold (EU AI Act Art. 15(1)-(2) / NIST MEASURE 2.5)
- Hallucination and factuality rate measured on a representative task sample (NIST AI 600-1 risk category / ISO Clause 9.1)
- Bias and disparate-impact testing across relevant demographic or use-case segments (ISO Annex A data-quality controls / NIST MEASURE 2.11)
- Adversarial red-team results covering prompt injection and jailbreak resistance (EU AI Act Art. 15(4)-(5) / NIST MEASURE 2.7)
- AI system impact assessment covering foreseeable misuse and affected stakeholders (ISO Clause 8.4 / EU AI Act Art. 9(2))
- Named sign-off from an accountable owner independent of the build team (ISO Clause 9.2 / NIST GOVERN 4.1)
- Post-release monitoring plan defining drift, incident, and re-evaluation triggers (EU AI Act Art. 72 / NIST MANAGE 4.1)
Frequently Asked Questions
What counts as evaluation evidence for an LLM before release?
It is the documented record showing a model was tested against defined metrics before deployment: accuracy and grounding results, bias and fairness testing, adversarial red-team findings, and a named sign-off. A benchmark score without a documented test plan, threshold, and owner is not evidence an auditor will accept.
Does NIST AI RMF require evaluation evidence before deployment?
NIST AI RMF is voluntary, but its MEASURE function specifically calls for AI systems to be evaluated for performance, fairness, and security before and after deployment. Many procurement teams and state regulators now treat AI RMF evidence as the practical floor even where it is not legally mandatory.
Which ISO 42001 clause covers testing an AI system before it goes live?
Clause 8.4 requires an AI system impact assessment, and Clause 9.1 requires monitoring, measurement, analysis, and evaluation of system performance. Together they form the evidence base an ISO 42001 auditor expects to see before certifying that an organization operates its AI management system as designed.
Does the EU AI Act require testing before a high-risk AI system is released?
Yes. Article 15 requires high-risk AI systems to reach an appropriate level of accuracy, robustness, and cybersecurity, tested against declared metrics, before they are placed on the market. Article 9 requires this testing to sit inside a documented risk management system, and Article 72 extends the obligation into post-market monitoring.
Is a general-purpose LLM automatically a high-risk AI system under the EU AI Act?
Not automatically. General-purpose AI models carry their own transparency and, for systemic-risk models, evaluation obligations under Articles 51 to 55. An LLM only inherits the Article 15 high-risk testing duties when it is deployed inside a use case listed in Annex III, such as employment screening or credit decisions.
How often should LLM evaluation evidence be refreshed?
Refresh evaluation evidence at every model version change, prompt or system-instruction change that affects behavior, and on a fixed schedule (commonly quarterly) even without a change, since underlying vendor models can drift. ISO 42001 Clause 9.1 and EU AI Act Article 72 both expect this to be a recurring control, not a one-time gate.
What is the difference between model evaluation and AI system impact assessment?
Model evaluation measures how the LLM performs against technical metrics like accuracy, bias, and robustness. An AI system impact assessment, required under ISO 42001 Clause 8.4, goes further and evaluates the consequences of deploying that model in a specific business process for affected people, the organization, and society.
Who should sign off on LLM evaluation results before release?
Sign-off should come from someone accountable for AI risk who is independent of the team that built or fine-tuned the model, consistent with NIST AI RMF’s GOVERN function and ISO 42001’s Clause 5 leadership requirements. Engineering test results alone, without an independent reviewer, are a common audit finding.
The Bottom Line
The three frameworks describe the same underlying discipline from three angles: NIST asks whether you measured it, ISO asks whether you assessed its impact and kept the record, and the EU AI Act asks whether you can prove it before a regulator does. Built once, a proper evaluation package answers all three.
Start with the release gate. If your organization cannot currently produce a five-category evaluation package tied to a specific production model version, that is the gap to close first, before adding more frameworks to track.
Govern365.ai, by the Global AI Certification Council, brings model registry, risk assessment, and audit evidence management into one workspace built around this exact evidence trail. Start your 14-day free trial to see how it maps your evaluation records to ISO 42001, the EU AI Act, and NIST AI RMF automatically.
