The AI governance checklist for enterprise teams
A working checklist of the governance controls every production AI system needs - evaluation harnesses, red-teaming, PII redaction, audit trails and policy-as-code - in the language your CISO will actually read.
Governance is the single most under-budgeted line in enterprise AI programmes, and the single most common reason a shipped system gets quietly pulled back out of production six months later. CISOs and audit committees are now reading AI system designs with the same rigour they read database designs. Getting governance right is not optional; getting it wrong adds a year to adoption.
This is a working checklist, written for the delivery lead or CISO partner who needs to sign off a design before it ships. Everything here has to exist as code and evidence, not as a paragraph in a slide deck.
1. Evaluation harness
Every production AI system needs an evaluation harness. Not an offline benchmark - a harness that runs in CI and in production.
Minimum viable:
- A golden set of 100–500 representative inputs with known expected outputs, maintained in version control.
- Automated scoring against that golden set on every merge.
- Weekly production sampling of real inputs with delayed human review, fed back into the golden set.
- A regression alert: if accuracy on the golden set drops by more than a threshold, the deploy is blocked.
Signs you don't have one yet:
- "We check it manually."
- The last golden-set update was more than 60 days ago.
- You have no way to catch a vendor model update breaking your outputs.
2. Red-team harness
A red-team harness is adversarial inputs run continuously against your system to catch prompt injection, jailbreaks, policy violations and edge-case behaviour.
Minimum viable:
- A catalogue of at least 50 adversarial probes, organised by attack class.
- Probes run nightly in staging, weekly in production.
- A defined severity taxonomy and a written escalation path.
- Coverage review quarterly - the attack surface changes.
This is the single area where most enterprise teams under-invest and it is the first thing an external auditor will ask for.
3. PII and PHI redaction
Any system that touches personal data needs an explicit redaction layer. Not "we told the model not to", a real layer that enforces it.
Minimum viable:
- Pre-inference redaction of PII/PHI from all logs and prompts.
- A configurable policy: what fields are redacted, with what replacement tokens, under what conditions.
- Post-inference validation that no redacted field leaked back into output.
- An audit record of every redaction event.
In regulated industries, this layer is usually the compliance case for deployment. It needs to be reviewable by someone who doesn't write Python.
4. Audit trail and provenance
Every AI-driven decision that affects a customer, a citizen or a financial transaction needs an audit trail.
Minimum viable:
- Input, model version, prompt template version, output, timestamp, latency, cost.
- Tool calls made, with inputs and outputs, if the system is agent-based.
- A queryable store retained for the compliance period (usually 7 years in finance, variable by jurisdiction in healthcare and public sector).
- Provenance - which document / source / knowledge base contributed to the output.
The audit trail should be indexed by customer / case / transaction ID so that if someone complains on day 180 you can reconstruct exactly what the system did on day 7.
5. Policy-as-code
The rules your AI system must follow - content policies, data access policies, escalation rules - should be code, not a PDF.
Minimum viable:
- A policy repo, versioned, with named owners per policy.
- Each policy enforced as a runtime check or pre/post-inference filter.
- A test suite validating that each policy is actually enforced.
- Change review by risk/compliance before policy updates ship.
Policy-as-code is how you avoid the death-by-memo pattern where the rules exist in a Confluence page that nobody quite follows.
6. Access control and isolation
AI systems often need access to sensitive data. The access pattern must be reviewable.
Minimum viable:
- Role-based access to any retrieval source.
- Per-request scoping of what the model can see (customer A's query cannot see customer B's data).
- Isolated inference: no model-call pool shared across tenants, in regulated deployments.
- Key management via your existing secrets infrastructure, not hardcoded.
7. Incident response and on-call
Something will go wrong. Governance is how quickly you notice and recover - usually the job of an embedded squad staffed by the people who built the system.
Minimum viable:
- A paged on-call rotation covering production AI systems.
- A written runbook for the top 10 failure modes.
- A post-incident review within 72 hours of any significant incident.
- A root-cause tracker reviewed monthly.
8. Human-in-the-loop design
For every AI decision, write down whether a human must, may, or does not need to be involved.
Minimum viable:
- A decision register listing every AI decision type in the system.
- For each, the human-in-the-loop policy and escalation path.
- Evidence that escalations are actually happening at the expected rate.
A system where the human-in-the-loop fires on 2% of decisions when the design said 15% is not well-governed. The escalation rate is one of the clearest signals of whether the design is working.
9. Model and vendor transparency
Know what you're running.
Minimum viable:
- A register of every model used in production with version, provider, and the task it's used for.
- A change process for model updates, with evaluation and red-team runs before deployment.
- A contractual record of data-handling commitments from every model provider.
10. Quarterly governance review
Governance drifts if it isn't reviewed. A quarterly review with risk, compliance and engineering keeps the system honest.
Minimum viable:
- Review of evaluation-harness results, red-team coverage, incident log and policy changes.
- A single attestation signed by the delivery lead.
- A documented outcome of what needs to change before the next review.
A note on frameworks
NIST AI RMF, ISO/IEC 42001, the EU AI Act - there is no shortage of frameworks. Pick the one that matches your regulator, and map the checklist above onto it. All of them ultimately require roughly the same controls; the difference is in reporting format and audit cadence.
Don't let framework selection slow shipping. The controls in this list satisfy most frameworks if they are implemented properly and evidenced.
Related reading
Frequently asked
What are the essential AI governance controls for enterprise deployment? Evaluation harness, red-team harness, PII/PHI redaction, audit trail, policy-as-code, access control, on-call response, human-in-the-loop design, model register, and quarterly review. All as code and evidence, not documentation.
How much does AI governance cost? Typically 15–25% of year-one programme cost in regulated industries. Less in consumer software; more in healthcare and public sector.
What framework should I use for AI governance? Map your regulator's expectations - NIST AI RMF, ISO/IEC 42001, or the EU AI Act as applicable - onto the checklist above. The underlying controls are the same; frameworks differ mainly in reporting.
Who owns AI governance internally? A joint responsibility between engineering, security, and risk/compliance, usually led by whoever owns the production system. A pure-compliance owner without engineering authority is a common failure pattern.
Safemode's governance service builds every one of these controls as code, wired into CI and production, with an audit pack your CISO can read cold.