The AI Governance Framework for Companies That Ship

A production AI governance framework is a seven-gate release system, not a policy binder. It should tell a delivery team what an AI system may access, what it may do, when a person must approve its work, how quality is measured, what evidence is retained, and who can stop or release it.

The practical test is simple: before an AI feature reaches users or business systems, can the team produce a signed release record covering data, permissions, human checkpoints, evaluation results, runtime alerts, fallback behavior, retention, and ownership? If not, governance is still an aspiration.

This guide turns that standard into seven production gates. Use them for an assistant that drafts responses, a retrieval system that answers from company documents, or an agent that can update business records. The controls become stronger as the consequences become harder to reverse.

The seven-gate production AI governance framework

Every AI system should pass seven gates before production:

Gate 1, outcome and ownership. Name the business outcome and the accountable owner.

Gate 2, data and model boundary. Define the approved sources, model path, retention, and revocation controls.

Gate 3, permission envelope. State what every tool may read, recommend, write, trigger, or escalate.

Gate 4, human checkpoints. Set approval points and a low-confidence fallback.

Gate 5, evaluation. Test a versioned evaluation set against release thresholds.

Gate 6, runtime control. Monitor behavior, retain evidence, and prepare containment.

Gate 7, release decision. Sign the release record and govern every material change.

The gates are deliberately operational. They can sit beside your product, security, and delivery process without pretending that one document solves every governance question.

The NIST AI Risk Management Framework organizes AI risk work through Govern, Map, Measure, and Manage. The production gates below translate that structure into evidence a team can inspect before and after release. The NIST Generative AI Profile provides additional considerations for generative systems. Neither replaces the decisions that belong to your organization and workflow.

Gate 1: name the outcome, owner, and affected people

Start with the business decision, not the model. A useful release record should answer five questions: What workflow is changing? What outcome should improve? Who owns that outcome? Who is affected when the system is right or wrong? What happens today when the system is unavailable?

Avoid an owner such as “the AI team.” Assign a person who can accept the operating decision, approve a release, fund fixes, and stop the system. Technical ownership and business ownership may sit with different people, but neither should be missing.

Define the workflow in verbs. A system may read approved documents, recommend a response, write to a CRM field, trigger a notification, or escalate an exception. Those verbs expose the real consequences. “AI assistant” does not.

The team should also record exclusions. A support assistant might answer product questions but never change account access. A sales agent might draft an email but never send it without approval. An operations tool might classify an invoice exception but never authorize payment.

Release evidence: one workflow statement, one accountable business owner, one technical owner, affected-user groups, expected outcome, and explicit exclusions.

Gate 2: define the data and model boundary

Governance fails quickly when the data path is vague. Map what enters the system, what can be retrieved, what leaves the organization, what is stored, and what may appear in an output.

For each data source, record the system of record, the fields or document collections the AI can access, the classification applied by your organization, and the permitted purpose. Add the model or service that receives the data, storage and retention behavior, the deletion or revocation path, and the owner who can approve a boundary change.

Do not describe the boundary only at vendor level. A hosted model may be permitted for public documentation but not for a customer record. A private deployment may still expose more data than the workflow requires. The control is the combination of source, field, purpose, model path, retention, and user.

For retrieval systems, maintain an approved-source register. Record which collections are allowed, how documents enter the index, how access rules are preserved, and how stale or withdrawn content disappears. Retrieval quality is part of governance because a model can give a fluent answer from the wrong source.

If private deployment or tighter infrastructure boundaries are required, the AI infrastructure service describes the delivery layer KUMO uses for production AI environments. The governance record should still state the exact boundary for the system being released.

Release evidence: a data-flow diagram, approved sources, model and hosting path, retention rules, access controls, and a tested deletion or revocation procedure.

Gate 3: set the permission envelope

An AI system should receive the narrowest capability required for its job. Write the permission envelope as a list of allowed and prohibited actions.

A useful envelope separates five levels:

Tool access should be explicit. List each API, function, database, queue, and file store. Specify the permitted operations and parameters. Separate testing and production credentials. Make every credential revocable without rebuilding the entire system.

Add safeguards outside the model. A prompt asking the model to “be careful” is not an authorization control. Use schema validation, field allowlists, transaction limits defined by the workflow owner, approval checks, and server-side enforcement.

Test whether the system refuses actions beyond its permission envelope. Include direct requests, indirect instructions found in retrieved content, malformed inputs, repeated calls, and retries after a timeout.

Release evidence: tool inventory, operation and field allowlists, credential ownership, refusal tests, duplicate-action controls, and rollback steps.

Gate 4: place human checkpoints and low-confidence fallbacks

“Human in the loop” is incomplete unless the team defines where the checkpoint sits, who receives it, what evidence they see, and what happens when they do nothing.

Place a checkpoint before actions that are difficult to reverse, affect a person, change sensitive records, or operate outside the system’s strongest evaluation coverage. Lower-impact work may use sampled review or post-action monitoring if the owner accepts that design.

A checkpoint should show the operator the proposed action, source evidence, escalation reason, relevant confidence or quality signals, and every field or system that will change. It should also expose clear approve, edit, reject, and defer paths.

Confidence alone should not decide whether a system acts. A high score can still be wrong, and scores are not always comparable across tasks. Combine model signals with deterministic conditions such as missing required data, conflicting sources, restricted actions, unusual volume, or an out-of-scope request.

Define the low-confidence fallback before launch. The system may ask for missing information, return a bounded response, route to a queue, use a deterministic rule, or stop. Silent guessing is not a fallback.

The human-in-the-loop AI guide explains how to match approval patterns to workflow impact. The production release record should add named owners, response targets, timeout behavior, and evidence shown to the reviewer.

Release evidence: checkpoint map, approver roles, review interface, escalation reasons, timeout behavior, and tested fallback paths.

Gate 5: measure quality before release and continuously

A polished demonstration is not a release test. Build an evaluation set from the actual workflow and its failure modes.

Start with normal, common cases, then add ambiguous or incomplete inputs and old, conflicting, or missing source material. Test requests that exceed permissions, sensitive-data cases, tool failures, timeouts, and repeated or duplicate actions. Include instructions embedded in retrieved content, cases that must be escalated, and examples representing the highest-impact failure.

Define metrics before seeing the result. Depending on the workflow, these may include task correctness, retrieval relevance, groundedness, refusal behavior, tool selection, parameter accuracy, escalation quality, duplicate-action rate, trace completeness, latency, and operating cost.

A single average can hide a dangerous tail. Set separate thresholds for critical cases. A system may perform well overall while failing the exact requests that carry the highest consequence.

Record the evaluation set version, model version, prompt or policy version, retrieval snapshot, tool version, results, known limitations, and reviewer. If the system changes, the team should know which tests must run again.

KUMO’s AI product and platform engineering service covers evaluation and production integration as part of the delivery system. The governance gate remains a decision owned by the organization releasing the workflow.

For an example of a production decision-support system, see the AutoIQ case study.

Release evidence: versioned evaluation set, thresholds, results by risk group, failure review, known limits, and signed acceptance criteria.

Gate 6: monitor behavior, retain evidence, and prepare response

Production governance begins at release. Define what the team will observe and what happens when a signal moves outside its accepted range.

Monitor input and output volumes, failures, refusals, escalations, retrieval and source errors, tool-call failures, and rejected parameters. Track human approval, edit, and rejection patterns alongside quality samples from live work, latency, operating cost, access or permission violations, and the active model, prompt, source, and integration versions.

Choose retention deliberately. Keep the evidence needed to investigate an incident, reproduce a decision, compare versions, and support the organization’s own obligations. Avoid retaining sensitive content by default when identifiers, hashes, structured event records, or redacted samples can provide the required evidence.

Create alerts with an owner and an action. “Quality dropped” is not enough. State the threshold, detection window, owner, immediate containment step, investigation evidence, communication path, and conditions for restoration.

Prepare rollback or containment before launch. Options include disabling a tool, moving the workflow to approval-only mode, reverting a model or prompt, removing a source collection, revoking credentials, or returning to the previous manual process.

The AI vendor exit and handover checklist provides acceptance tests for access, environments, observability, backup, rollback, ownership, and continuity. Those controls make governance durable when people or vendors change.

Release evidence: monitoring specification, alert thresholds, owner schedule, retention record, incident path, and tested containment or rollback.

Gate 7: sign the release and govern every material change

The final gate converts evidence into a decision. A release review should include the business owner, technical owner, and the people responsible for data, security, and operations where relevant to the workflow.

Use four possible outcomes:

Release. All required evidence meets the accepted thresholds.

Release with limits. The system may operate for a defined scope, user group, or permission level while named follow-up work remains.

Return for remediation. Evidence is incomplete or a threshold has not been met.

Stop. The workflow should not proceed under the proposed design.

Record the decision, evidence links, known limits, owner, date, and next review trigger. A decision without a named owner and conditions is only a meeting note.

Define material changes before launch. A model replacement, new data source, new tool, broader permission, new user group, changed retention behavior, altered approval boundary, or significant prompt and policy change may require partial or complete reevaluation.

Do not force every small copy edit through a full release board. Use a change matrix that maps each change type to required tests and approvers. The goal is traceable control without making safe iteration impossible.

Release evidence: signed release record, operating limits, change matrix, review triggers, and the next scheduled governance review.

The release packet your team should be able to produce

A production AI release packet should fit in one navigable evidence set:

1. Workflow record. Capture the outcome, owners, affected people, and exclusions.

2. Data record. Capture the data flow, approved sources, model path, retention, and revocation.

3. Permission record. Capture the tool inventory, permission envelope, credentials, and rollback.

4. Human-control record. Capture checkpoints, fallback, queues, and timeout behavior.

5. Evaluation record. Capture the evaluation set, thresholds, versioned results, and known limits.

6. Runtime record. Capture monitoring, alerts, retention, incident response, and containment.

7. Release record. Capture the decision, operating limits, change matrix, and review date.

The packet does not need to be one document. It may link to architecture diagrams, test results, access policies, runbooks, and dashboards. What matters is that the evidence is complete, owned, current, and usable during a real release or incident.

Start with one consequential workflow

Do not begin by writing governance language for every possible AI use. Choose one workflow that matters, map its production boundary, and use the seven gates to expose missing decisions.

A strong first workshop should leave with a named owner, data boundary, permission envelope, human checkpoint, evaluation plan, monitoring plan, and release path. That is enough to turn governance from policy intent into delivery work.

Book a 30-minute discovery call to map the release gates for one production AI workflow.

Sources

NIST AI Risk Management Framework

NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

RAND, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed