AI Model Governance: Inventory, Model Cards, Re-Validation

What Is AI Model Governance, From Registration to Retirement?

AI model governance is the set of records, named owners and checks that decides which AI models an organization may use, at which exact version, for which purpose, how each is validated before first use, when it must be re-validated, and how it is retired. It applies to models you train, open-weight models you download and hosted models you only reach through an API.

The unit is one model version in one use. That is narrower than an AI governance framework, which sets organization-wide principles, risk tiers and committees, and narrower than agentic AI governance, which adds controls for systems that act through tools.

NIST's AI Risk Management Framework (NIST AI 100-1) brackets an AI system's life with two neighbouring GOVERN subcategories, and both apply to each model the system uses: GOVERN 1.6 asks for mechanisms to inventory AI systems, and GOVERN 1.7 asks for processes "for decommissioning and phasing out AI systems safely". Between those two points sit the activities this guide covers:

• Register - give every model version an inventory entry with an owner before any system calls it.
• Document - read the publisher's model card and record what it leaves out for your use.
• Validate - test the model in the system that will call it, on your own data, and keep the evidence.
• Re-validate - repeat the same checks whenever the model, its prompt, its tools or its use change.
• Retire - move every caller off a version before its provider switches it off, and keep the record.

What Does an AI Model Inventory Need to Record?

An AI model inventory needs enough fields to answer, for any model version, who uses it, for what, on whose approval, and with what evidence. Annex IV of the EU AI Act (Regulation (EU) 2024/1689) asks the technical documentation of a high-risk system for "the version of the system reflecting its relation to previous versions" and for any "recourse to pre-trained systems or tools provided by third parties and how those were used, integrated or modified by the provider".

Kept as a file next to the code, one inventory entry can look like this example record for a hosted model (the model name, dates and paths are illustrative):

id: mdl-0042
model: example-chat-model
version: "2026-05-14"      # dated version, never an alias
provider: Example Cloud
access: hosted-api         # hosted-api | open-weights
base_model: null
license: provider terms, read 2026-09-30
owner: card-disputes product team
validator: model risk team (independent)
intended_use: draft replies to card disputes
excluded_uses: [credit decisions, fraud scoring]
used_by: [dispute-agent, help-center-bot]
risk_tier: high
upgrade_setting: no-auto-upgrade
provider_retirement: "2027-11-14"
last_validation: "2026-09-30"
evidence: evals/dispute-agent/2026-09-30/
revalidate_on: [new-version, prompt, tool,
  retrieval, fine-tune, new-use, incident]
status: approved

The "used_by" list does the most work, because it is the list of systems one retirement notice will touch. Without it, a provider's notice reaches whoever owns the cloud account, and nobody can say which products are affected.

What Should a Model Card Contain?

The format comes from Model Cards for Model Reporting by Margaret Mitchell and colleagues (arXiv, 2018; FAT* 2019), which describes model cards as "short documents accompanying trained machine learning models" that provide "benchmarked evaluation in a variety of conditions". On the Hugging Face Hub, the card is the README.md of the model repository, with a YAML metadata header on top.

The Hugging Face model card documentation says a card should describe the model, its intended uses and potential limitations, the training parameters, the training datasets and the evaluation results. Read against governance needs, each part answers a different question:

• License - the license field, or "other" with a named license and link; it decides whether your intended use is allowed at all.
• base_model - the model a fine-tune, adapter, quantization or merge was derived from, which is the lineage the inventory needs.
• datasets - the training datasets by Hub ID, the starting point for data provenance and copyright questions.
• Evaluation results - scores in structured metadata, either the older model-index block or YAML files in a .eval_results/ folder, which the Hub documentation introduces as a newer, simpler format.
• Intended uses and limitations - prose sections on what the publisher designed the model for, its known biases and the uses it advises against.

Hugging Face Hub, top 100 text-generation models
by downloads, read 2026-10-19
Publishing accounts: 36
==================================================
Card metadata (YAML header)
  license declared                    94 of 100
  license field set to 'other'        15 of 100
  base_model declared                 51 of 100
  training datasets declared          13 of 100
  eval results (model-index)           2 of 100
  eval results (.eval_results/)        7 of 100
README section headings
  limitations, bias or risks          27 of 100
  evaluation or benchmarks            68 of 100
  intended use or how to use          31 of 100
Access
  gated (accept terms first)           4 of 100
  README under 1,000 characters        3 of 100
==================================================
All five: license, base, data,         0 of 100
  eval results, limits section

How Do You Govern Models You Did Not Train?

You govern a hosted model through its provider's lifecycle policy, because the provider decides when your version stops working. Microsoft's Foundry Models lifecycle and support policy, read on 2026-10-19, gives a standard 18-month lifecycle for generally available models: new customers lose access at 12 months, at least 60 days of notice precede retirement, and after retirement "all inference returns 410 Gone".

The same policy shows why the upgrade setting belongs in the inventory. Global Standard, Data Zone Standard and Standard deployments are upgraded automatically, region by region, when a version retires; provisioned deployments are not, and a deployment set to NoAutoUpgrade stops working at retirement. Microsoft also advises teams to "Compare quality, latency, and cost together rather than relying on public benchmarks alone."

For hosted models, put these controls in place before a version change reaches users:

• Pin a dated version - call the model by its dated version identifier, never by an alias that the provider repoints to newer releases.
• Record the retirement date - copy the provider's deprecation and retirement dates into the inventory and alert the owner well before the first one.
• Choose the upgrade setting on purpose - an automatic upgrade swaps the model without a release; no upgrade means an outage at retirement; either way, write the choice down.
• Validate the candidate early - run your evaluation set on the replacement as soon as it is available, not in the last week.
• Keep a fallback - name a second approved model or a non-AI path for each high-risk use.
• Ask for the documentation you are owed - under Article 53 of the EU AI Act, providers of general-purpose AI models must give the providers that integrate them information on the model's capabilities and limitations, unless the model is released under a free and open-source licence with its weights, architecture and usage information public and carries no systemic risk.

The fallback step follows GOVERN 6.2 in the same NIST framework, which asks that "Contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be high-risk". For open-weight models the provider cannot retire your copy, but the same discipline applies: pin the repository commit, store the weights you validated, and treat a new upload under the same name as a new version.

How Should Validation and Re-Validation Work Across the Model Lifecycle?

Validate before first use, then re-validate on a defined list of changes rather than on a calendar alone. The supervisory guidance attached to SR 26-2 says "Validation generally occurs prior to a model's first use" and splits validation into conceptual soundness, outcomes analysis and ongoing monitoring; for vendor models it calls for "ongoing monitoring and outcome analysis to assess whether vendor models are accurate, remain fit for purpose, and continue to be reliable".

Write the triggers into the inventory so the re-validation decision is mechanical:

Score old and new versions on the same reviewed, versioned items, as described in the guide to building a golden dataset, and compare them item by item rather than by headline pass rate, the approach set out in LLM regression testing.

When a high-risk system falls under the EU AI Act, plan model updates before the first assessment. Article 43(4) requires a new conformity assessment after a substantial modification, but for a system that keeps learning after it is placed on the market, changes the provider pre-determined at the initial assessment and recorded in the technical documentation do not count as one.

What Do Regulators Expect From Model Governance?

The sources below differ in force: one is binding EU law, one is voluntary US guidance and one is US banking supervision guidance. The dates come from each source as read on 2026-10-19; for the EU AI Act, the application dates are from the European Commission's AI Act policy page, last updated on 19 October 2026, which reflects the AI Omnibus that entered into force on 27 July 2026.

The Federal Reserve's SR 26-2 letter says the revised guidance supersedes SR 11-7, issued on 4 April 2011, and SR 21-8, and reflects supervisory experience and industry feedback "accumulated over the past fifteen years". It adds: "This letter is expected to be most relevant to banking organizations with over $30 billion in total assets regulated by the Federal Reserve." The attached guidance then carves newer models out in a footnote: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."

The same footnote adds that "a banking organization's risk management and governance practices should guide the determination of appropriate governance and controls" for systems it does not cover. A bank's large language models therefore still need governance; the banking agencies' guidance just does not prescribe it, so the bank's own policy has to.

The EU AI Act is narrower than it first looks for teams that only call a model: most of its model-level duties fall on the provider of the general-purpose model or of the high-risk system. Timing also differs for models already on sale: under Article 111(3), providers of general-purpose AI models placed on the market before 2 August 2025 have until 2 August 2027 to comply. How to turn its risk management article into test cases is covered in EU AI Act conformity testing.

Who Owns AI Model Governance?

Ownership splits across people who use, build, check and audit a model. The banking agencies' revised model risk guidance warns about conflicts "between different reporting lines, such as model development and validation groups" and says internal audit's role "is generally to evaluate whether the model risk management practices are rigorous and effective", rather than to repeat validation.

Under the EU AI Act, roles can also move once the high-risk rules apply. Article 25 treats a deployer that makes a substantial modification to a high-risk AI system as its provider, with the provider's obligations, and its paragraph 4 requires the provider of a high-risk AI system and a third party that supplies an AI model used in it to specify, "by written agreement", the information, capabilities and technical access the provider needs to comply.

Re-Validating an AI Agent After a Model Change With Agent Assurance

Once the model behind an agent picks its tool calls and record writes, re-validation evidence has to cover the actions the agent took with the new model as well as its replies. TestMu AI built Agent Assurance to test such agents before they ship. It does not register, inventory, govern or certify models; the evidence it produces is something your model owner and validator can attach to the inventory entry.

Rook CLI, the client you drive it from, is a pre-alpha release that is publicly installable through npm:

npm install -g @testmuai/rook

The rook command runs on macOS and Linux, and 64-bit Windows through npm or WSL. Installing through npm needs Node.js 20 or newer, while the Homebrew and shell installer routes for macOS and Linux, described in the Rook CLI installation docs, need no Node.

Each step of a model change has a counterpart in the Rook CLI workflow that the README in the public rook release repository describes:

• Generate the suite once - Agent Assurance reads the agent's codebase or spec and drafts scenarios with acceptance criteria attached; by default the set mixes functional and adversarial cases.
• Add a profile per model - an agent can hold several profiles, so the model in production and the candidate become two configurations of one agent.
• Run the same suite on both - point both runs at staging: whatever the agent writes during a run really happens, and nothing is undone afterwards.
• Read the run-over-run change - run the candidate profile right after the current one so the last two runs form the pair, check each run's profile, then read the pass-rate change and the scenarios that newly fail or were fixed; edited scenario definitions are flagged.
• Gate on the report - rook ends a finished run with exit code 0 whatever the verdicts, so a CI job should read them from rook report --json.

Every acceptance criterion closes as Pass, Fail or Unable to Verify, judged from the files a run changed, the agent's tool calls checked against the tools it declares, and read-only checks, through a tool you approve, on the records the run touched. Model judges score some criteria, and an agent saying it finished a task never counts as proof. A criterion with no evidence behind it is reported as Unable to Verify, and the scenario pass rate counts only scenarios with a decided verdict, so gaps stay visible.

Most eval and observability tools score what an agent said and recorded. Agent Assurance checks what the run changed and reports what it could not verify. It tests and reports; whether your model governance meets the EU AI Act, banking guidance or your own policy is for your compliance team and auditors to decide, and run evidence is stored as plain files, so use synthetic or staging data.

Starting AI Model Governance With a Model Inventory and a Re-Validation Run

Start with a search of your code and configuration for every model identifier, and give each one an inventory entry with its exact version, its callers and its provider's retirement date. For each high-risk use, read the model card, note which of the five basic fields are missing, and record the upgrade setting.

Retirement dates can come from the provider's own data rather than a forwarded email. Microsoft's Models - List API reference (API version 2024-10-01, read on 2026-10-20) returns each model's lifecycleStatus, its deprecation.inference date, which Microsoft's lifecycle policy maps to the version's retirement, its fine-tuning deprecation date, and a deprecationDate for each SKU, so a scheduled job can keep the inventory's lifecycle dates current.

Then pick the system whose model retires first and build its re-validation run before the notice arrives. If that system is an agent that acts, the Agent Assurance CI/CD guide covers running a reviewed suite in CI under a chosen profile and gating the job on the report's verdicts.

Note: This article was researched and drafted with AI assistance, then reviewed, fact-checked, and published by Sanjay Singh, Senior Product Manager at TestMu AI and the article's author of record, whose listed expertise includes AI/ML products and FinTech. Sirajuddin Khan, Vice President of Product Management at TestMu AI, reviewed it. Every statistic, link, and product claim was verified against primary sources: the EU AI Act text, NIST, the Federal Reserve, Hugging Face documentation and API data, provider documentation, research papers and TestMu AI's product documentation. Read our editorial process and AI use policy for details.