AI Vendor Exit Checklist: 10 Production Handover Tests
Use 10 acceptance tests to verify code, cloud, data, models, access, monitoring, rollback, and ownership before completing an AI vendor handover safely.
Aug 6, 2026
The sign-off rule
An AI vendor handover passes only when the incoming owner can make a controlled change, deploy it, observe the result, restore service, and explain how the AI behaves without depending on the outgoing vendor.
Receiving a repository and a folder of documents is not enough. A production AI system also depends on cloud accounts, secrets, model settings, prompts, retrieval data, evaluation cases, safety rules, monitoring, cost controls, incident history, and the business decisions behind exceptions.
Use the 10 acceptance tests below before ending a retainer, releasing a handover milestone, changing agencies, or moving the system to an internal team. Treat failed ownership, deployment, rollback, data, access, or evaluation tests as sign-off blockers.
The 10-test handover checklist
Use this table to identify the evidence required before sign-off. A critical failure in ownership, deployment, recovery, data, access, or evaluation remains a blocker even if other evidence is ready.
| Test | Evidence required for sign-off |
|---|---|
| 1. Production ownership | Company-controlled repositories, cloud, domains, data stores, billing, and vendor accounts |
| 2. Reproducible deployment | The incoming owner deploys a small approved change through the real release path |
| 3. Rollback and recovery | A rollback and one non-production backup restore succeed |
| 4. AI configuration | Models, prompts, tools, retrieval rules, fallbacks, and versions are recorded |
| 5. Evaluation baseline | A reusable test set runs before and after the handover with comparable results |
| 6. Data control | Sources, permissions, retention, deletion, and regional requirements are mapped |
| 7. Access and safety | Named access, least privilege, approvals, audit trails, and secret rotation are verified |
| 8. Monitoring and cost | Dashboards, alerts, incident history, usage, and spend baselines have named owners |
| 9. Product context | Business rules, exceptions, support themes, accepted risks, and roadmap decisions are recorded |
| 10. Independent operation | The incoming team handles a change and a simulated incident without the outgoing vendor operating the tools |
If you need an independent review before changing vendors, book a 30-minute discovery call. KUMO will map the failed tests, production risks, and smallest safe transition plan.
1. Confirm company ownership of every production dependency
Start with control, not documentation. Your organization should administer the production repositories, cloud accounts, domains, databases, vector stores, model-provider accounts, monitoring tools, CI/CD systems, billing, and support systems.
A shared password is not ownership. Use named accounts, role-based permissions, multifactor authentication, and a current access register. Record any component that must remain under a third-party account, why it cannot move, who owns the commercial relationship, and how the system will operate if that provider changes.
Pass when: the incoming owner can reach every production dependency through company-approved access, and no undisclosed vendor-owned account is required to keep the system running.
For a broader non-AI transfer, use KUMO's agency-to-in-house engineering handover checklist.
2. Make the incoming owner deploy a real change
Documentation can describe a release process that no longer works. The only reliable test is for the incoming owner to use the actual pipeline.
Choose a small, reversible change. The incoming owner should create the branch, run automated checks, deploy to staging, run the AI evaluation set, promote the release through the production path, and confirm that the expected dashboards and alerts respond.
The outgoing vendor may answer questions, but should not type the commands or approve hidden steps. Record every missing permission, manual dependency, undocumented environment variable, and decision discovered during the exercise.
Pass when: the incoming owner ships the approved change without the outgoing team operating the tools.
3. Prove rollback and backup recovery
A successful deployment proves only the forward path. The handover also needs evidence that the team can recover when a release, integration, data sync, or model change fails.
Run one controlled rollback to the previous application version. Restore one backup into a non-production environment and verify that the restored data is usable. For retrieval systems, confirm that indexes can be rebuilt from documented sources rather than copied from an unexplained vendor snapshot.
A useful recovery record names the recovery objective, commands, permissions, dependencies, expected duration, validation checks, and owner. KUMO's AI incident response runbook provides a related operating structure for customer-facing workflows.
Pass when: rollback and restore are demonstrated, timed, and repeatable by the incoming owner.
4. Inventory the AI decision layer
A software inventory lists services and integrations. An AI inventory must also explain how the system produces and constrains outputs.
Record every production model and version, the reason it is used, model parameters, system prompts, prompt templates, tool permissions, structured-output rules, retrieval sources, chunking rules, embedding models, index refresh schedules, safety filters, approval thresholds, fallbacks, and human escalation paths.
Version these items with the release that uses them. A routine provider or prompt change can alter quality, latency, cost, or safety even when the application code is unchanged.
Pass when: the incoming owner can reproduce the current AI configuration and identify which version produced a given output.
5. Preserve the evaluation baseline
A missing evaluation baseline can block a safe handover even when the code and infrastructure are complete.
The incoming team should receive representative evaluation cases, expected outputs or scoring rules, the current production results, separate thresholds for quality, safety, latency, and cost, known failure examples, and a process for adding cases after incidents or user feedback.
Run the same evaluation before and after the transition. If results move, the team should be able to isolate whether the cause is code, data, model version, prompt configuration, retrieval, or infrastructure. A polished demo with selected prompts is not a production evaluation.
NIST's AI Risk Management Framework and Generative AI Profile are useful external references for governance, measurement, testing, and ongoing risk management.
Pass when: the incoming owner can run the evaluation suite and explain any material difference from the agreed baseline.
6. Map data rights, retention, and deletion
AI systems often copy or transform data across application databases, warehouses, model APIs, logs, caches, vector indexes, evaluation sets, and support tools.
For each flow, record the source, destination, purpose, owner, sensitive fields, permissions, retention period, encryption controls, model-provider use, training settings, deletion process, backup behavior, and regional requirement.
Test deletion across both primary storage and AI retrieval systems. Removing one database row is not enough if the content remains in an index, log, cache, evaluation file, or provider account.
Pass when: the incoming team can trace a sensitive record through the system and execute the agreed deletion procedure.
7. Rebuild access around named owners and least privilege
Do not inherit the outgoing vendor's access model without review. Remove dormant users, replace shared credentials, rotate production secrets, and separate human access from service credentials.
High-impact AI actions need explicit limits. Document which tools can read, recommend, write, approve, or trigger external actions. Keep human approval for actions with financial, legal, safety, or customer-impact consequences. Preserve audit logs that connect an action to the user, service, model configuration, and release.
CISA's Secure by Design guidance and the OWASP Top 10 for LLM Applications provide useful security checks for access, sensitive information, excessive agency, model behavior, and downstream handling. KUMO's AI governance checklist for CRM and ERP workflows turns related controls into an operating review.
Pass when: every production identity has a named owner, justified permissions, a revocation path, and auditable activity.
8. Transfer monitoring, incidents, and cost baselines
The incoming team needs to know what normal production behavior looks like before it can spot a regression.
Transfer quality, reliability, performance, and cost baselines. Useful signals include task success, grounded-answer rate, approval and escalation rates, application errors, timeouts, queue depth, failed integrations, response time, throughput, model spend, infrastructure spend, and cost per completed workflow.
Also transfer recent incidents with symptoms, root cause, temporary action, permanent action, customer impact, and unresolved risk. A dashboard without alert ownership is incomplete. Every alert needs a threshold, recipient, response expectation, and escalation path.
Pass when: the incoming owner receives a real alert, finds the relevant logs, and explains the expected response.
9. Capture product context, not only technical state
A technically accurate handover can still fail if the new team does not understand why the system behaves as it does.
Record business rules, high-value user journeys, exceptions, approval policies, support themes, rejected options, accepted risks, unresolved defects, roadmap decisions, and the next 30 days of work. Link each important decision to the product or operating outcome it protects.
This context prevents a new team from removing an intentional constraint, repeating an abandoned experiment, or treating a known business exception as a technical bug.
Pass when: the incoming owner can explain the major exceptions, current risks, and next priorities to a business stakeholder without the outgoing vendor present.
10. Test independent operation with a controlled incident
The final test combines the previous nine. Give the incoming team a small approved change and a safe simulated failure. Ask them to diagnose the issue, use the dashboards and runbooks, choose a response, communicate the impact, and recover the system.
Do not grade presentation quality. Grade whether the team found the right evidence, used the correct access, protected customer data, made a reversible decision, and left an auditable record.
If the system is already unstable, run a short stabilization phase before switching ownership. KUMO's software project rescue plan explains how to separate recoverable delivery problems from cases that may require a rebuild.
Pass when: the incoming team handles the exercise while the outgoing vendor observes rather than operates.
A practical 14-day transition plan
The calendar is not the acceptance criterion. Use it to sequence ownership and evidence.
| Timing | Primary work | Exit condition |
|---|---|---|
| Days 1 to 3 | Freeze unplanned changes, confirm accounts, inventory dependencies, name owners | Every critical dependency and access gap is recorded |
| Days 4 to 7 | Walk through architecture, AI configuration, data flows, evaluations, incidents, and costs | Evidence is transferred and unanswered questions become tracked actions |
| Days 8 to 11 | Incoming team deploys, rolls back, restores a backup, rotates secrets, and runs evaluations | The incoming owner operates the real tools successfully |
| Days 12 to 14 | Run the controlled incident, close blockers, agree support boundaries, record accepted risks | Critical tests pass and remaining work has named owners and dates |
Complex or regulated systems may need longer. Extending the calendar without explicit evidence does not reduce transition risk.
Put handover requirements in the contract before work starts
The easiest exit is designed before the build begins. The statement of work or master agreement should define company ownership of project-specific IP, repositories and cloud accounts, current architecture and operations documentation, exportable AI configuration and evaluation assets, named handover deliverables, access revocation, secret rotation, and a bounded support period.
Tie acceptance to evidence, not the number of knowledge-transfer calls. Use KUMO's custom software QA and release checklist to align release evidence with operational acceptance, and the AI product maintenance plan to define post-handover ownership.
Decide whether to switch, stabilize, or rebuild
Switch now when the company lacks control of production assets, unresolved security or data risk remains, the vendor refuses basic handover evidence, or delivery depends on one unavailable person.
Stabilize first when production has active incidents, backups or rollback are untested, the incoming team lacks essential access, or there is no evaluation baseline for comparison.
Rebuild only after diagnosis when critical code or infrastructure cannot be recovered, the architecture cannot meet current requirements, or irreplaceable vendor-owned dependencies make safe operation impossible. A difficult relationship alone is not a technical reason to rebuild.
How KUMO supports an AI handover
KUMO's custom software development team can assess a vendor-built AI product across production ownership, deployment, integrations, evaluation, monitoring, and operating control. For a transition or rescue engagement, the first deliverable is an evidence map of the 10 tests above, not a promise to replace every component.
See how KUMO designed and built the AutoIQ decision-support platform.
The goal is to establish control, preserve what works, and make the next decision measurable. If you want that independent assessment, book a 30-minute discovery call.
