RAG Knowledge Base Acceptance Tests Before Employee Rollout

RAG Knowledge Base Acceptance Tests Before Employee Rollout

A RAG knowledge base needs 10 acceptance tests before employee rollout: identity, permission, source, freshness, retrieval, citation, refusal, deletion, escalation, and change regression.

A convincing answer in a demo does not prove that a knowledge assistant can serve a real team. The release decision turns on whether the assistant retrieves the right evidence for the right person, shows where the answer came from, refuses when evidence is insufficient, and forgets material that should no longer be available.

Use these 10 tests to decide whether one employee knowledge assistant should launch, stay limited, or stop. Each test names the evidence to collect, the failure boundary, and the next action.

The decision utility

Use the table as a release contract. Choose the intended mode before the test run, keep the same evidence pack while fixes are made, and change the mode only when the failed condition is actually resolved. A different slide deck or a smoother demonstration is not new evidence.

How to run the acceptance review

Start with one workflow, one user group, one source boundary, and one consequence. Record the exact configuration being tested, then run the 10 decisions below against normal work and the failures that would matter most. Keep source material, expected outcomes, and observed results separate so a later change can be compared honestly.

Test 1: Identity is established before retrieval

Every request must resolve to a real user, team, and tenant before search begins. Shared links, anonymous sessions, and copied chat URLs should not bypass the identity boundary. The test pack should include valid users, suspended users, contractors, and people who changed teams.

The evidence is an identity trace that connects the request to the retrieval filter and final answer. Pass only when the same question produces the permitted result for each role and no result for an ineligible role.

Block rollout if the system searches first and filters later, if cached answers cross roles, or if a removed user can continue a conversation. Fix the authorization sequence, expire stale sessions, and rerun the full role matrix.

Test 2: Document permissions survive indexing

A RAG index is another access path to company material. The source permission model must travel into chunks, metadata, retrieval filters, and citations rather than disappearing during ingestion. Folder membership alone is not enough when individual records have narrower rules.

Build paired tests with identical wording stored in public, team-only, executive, and customer-specific documents. Pass when each user receives only the records already available through the source system and the retrieval log explains every exclusion.

Hold the release if broad service credentials flatten access or if permission changes require an unbounded reindex. The implementation brief should name the authority source, sync delay, revocation path, and behavior while permissions are being updated.

Test 3: The source set has an accountable owner

The assistant cannot be more reliable than the material it is allowed to search. Every source collection needs an operational owner who can approve inclusion, remove obsolete content, resolve conflicts, and identify the system of record.

The acceptance packet should map each collection to purpose, owner, update method, retention rule, and authoritative status. Pass when a person can answer why each source is present and what happens when two documents disagree.

Hold when the corpus is a convenient folder dump, when duplicate policies compete without precedence, or when nobody can retire obsolete guidance. Curate the set before tuning retrieval because ranking cannot repair an unowned source base.

Test 4: Freshness is visible and testable

A correct old answer can still create a bad business decision. Freshness should be treated as part of answer quality, especially for policies, inventory, product instructions, customer status, and operating procedures.

Seed documents with known effective dates and superseded versions. Pass when the assistant prefers the current source, identifies the effective date where relevant, and warns when the index may lag behind the source system.

Limit use when the source is accurate but synchronization is delayed. Stop when the assistant silently cites retired policy. The fix may be event-driven updates, a shorter refresh interval, query-time retrieval, or a visible freshness warning.

Test 5: Retrieval finds evidence for real work questions

Synthetic questions written from document headings make retrieval look easier than daily work. Employees use abbreviations, incomplete context, customer language, and follow-up questions that cross document boundaries.

Create questions from resolved tickets, operating checklists, onboarding sessions, and expert interviews, then remove private identifiers. Pass when the retrieval set contains the evidence an experienced employee would use, not merely a semantically similar paragraph.

Hold when the answer sounds plausible but the retrieved material would not support the action. Improve chunking, metadata, query rewriting, or source selection, then rerun the unchanged question set to preserve comparison.

Test 6: Citations support the exact claim

A citation is useful only when it points to text that supports the sentence beside it. A link to a long handbook or a related page can create false confidence without providing traceability.

Pass when each material claim maps to a specific source segment, the employee can open that source, and the quoted context supports the wording. Include multi-source answers and conflicting sources in the tests.

Stop if a citation is inaccessible, unrelated, or generated from content the user cannot open. Do not hide the defect by removing citations. Fix the retrieval and answer contract so evidence and wording remain aligned.

Test 7: The assistant refuses unsupported requests

A useful knowledge assistant must distinguish missing evidence from a negative answer. When the source set does not contain the answer, the safe output is a bounded refusal or a request for more context, not a confident completion.

Include out-of-scope policies, nonexistent customers, future product features, ambiguous abbreviations, and questions that require professional judgment. Pass when the response states the limit and routes the employee to the right next step.

Block release when unsupported questions receive invented specifics or when the assistant blends public knowledge with private policy without disclosure. Strengthen the instruction, retrieval threshold, and response validator together.

Test 8: Deletion removes future access

Deletion must reach the source, index, cached answer, conversation history, derived memory, and any exported evaluation data that should no longer retain the content. Removing the original file alone may leave searchable fragments behind.

Create a uniquely identifiable test document, retrieve it, remove it through the normal business process, and verify every downstream store. Pass only when new searches and continued conversations cannot reproduce the removed content.

Stop the rollout if deletion is unverifiable or depends on an operator remembering several hidden stores. Document the deletion chain, automate confirmation where possible, and retain an audit result without retaining the deleted content.

Test 9: Uncertain answers reach a person with context

Employees need a clear path when the assistant cannot answer, sources conflict, or the requested action exceeds its scope. Escalation should carry the question, retrieved evidence, uncertainty, and user context without forcing the next person to reconstruct the session.

Pass when the system creates a bounded handoff to the right team, preserves permitted context, and tells the employee what will happen next. Test unavailable queues, after-hours use, and escalation rejection.

Hold when escalation is a generic email address or when private conversation content is copied into a wider channel. Define destinations by issue type and apply the same access controls to the handoff payload.

Test 10: Changes rerun an unchanged regression set

New documents, embedding changes, prompt edits, model updates, and retrieval tuning can improve one question while breaking another. The rollout needs a stable evaluation set that survives implementation changes.

Pass when every material change reruns the same representative questions and records retrieval, answer, citation, refusal, latency, and access outcomes. Add new failures to the set without rewriting old cases to make the result easier.

Hold a change when a critical case regresses, even if average quality appears better. The decision should be tied to named release rules and business consequences rather than a single aggregate score.

Worked operating scenario

A service company wants employees to search operating procedures and customer implementation notes. The safest first release includes the current procedure library for one delivery team, excludes customer-specific notes until tenant filters pass, and keeps policy exceptions in a human queue. The team can expand only after deletion and role-change tests pass on the same evidence pack.

The scenario is deliberately narrow. It shows how a useful first release can preserve business value while limiting authority. Expansion should follow evidence from the same acceptance contract, not pressure to include more document types, teams, systems, or actions in the first milestone.

What the implementation brief must contain

The implementation brief should name the initial user group, source systems, authoritative collections, permission authority, indexing method, freshness expectation, citation format, refusal behavior, escalation destination, deletion chain, evaluation questions, release modes, and operator. A vendor proposal that cannot identify these items is still describing a demo rather than an employee rollout.

Ask the delivery team to return the brief as an acceptance map rather than a feature catalogue. Each item should identify the expected evidence, the condition that blocks release, the person responsible for the decision, and the recovery path. This makes proposal comparison more concrete and makes handover possible later.

Connected decisions in the KUMO library

Use the RAG vs AI agents guide to choose between retrieval and action before you test a knowledge assistant for rollout.

Data strategy for AI helps define source ownership, lineage, quality, and lifecycle before documents enter the knowledge base.

Use the AI agent security checklist when the system can also take actions or reach tools, accounts, and customer data.

The AI readiness assessment helps determine whether the organization has the process, data, and ownership needed to start.

After release, the AI workflow monitoring dashboard guide helps track answer quality, latency, cost, and human escalation.

What to do this week

Choose one real workflow and write the release mode you want. Then select representative work, boundary cases, and one failure that would cause the most business harm. Run the first evidence review before adding integrations or expanding scope.

Next, assign one person to own the operating result and another to challenge the evidence. The goal is not a committee. It is to prevent the builder from being the only person deciding whether the build passed. Record disagreements as missing acceptance criteria and resolve them in the contract.

Finally, rehearse the stop path. Disable the feature, preserve the permitted evidence, tell the affected user what happened, and restore service only after the release condition passes. A workflow that cannot stop safely is not ready to gain more authority.

A practical route from decision to build

KUMO builds production AI and custom software for growing businesses. The AI integration service is the route for turning this acceptance map into a scoped implementation. The Volopay case study shows how KUMO approaches a production software build.

If you want to turn the evidence map into a scoped first release, Map the first RAG rollout milestone.

Primary sources used for this framework

NIST Generative AI Profile frames risks such as confabulation, data privacy, and human oversight that acceptance tests should address.

OpenAI file search guide documents how files enter vector stores and how retrieval results can support generated answers.

Amazon Bedrock knowledge bases documents retrieval, citations, data sources, synchronization, and access options for managed RAG systems.

OWASP Top 10 for LLM applications covers risks such as prompt injection, sensitive information disclosure, and vector or embedding weaknesses.

Microsoft RAG overview describes retrieval, grounding, citations, content preparation, and access controls in Azure AI Search.

Implementation decisions to resolve

What is the minimum scope for a RAG rollout?

Use one named employee group, one curated source collection, one identity boundary, and one set of work questions. Expand only after access, freshness, citations, refusal, deletion, and regression evidence pass for that scope.

How should a RAG assistant refuse unsupported requests?

No. It should state that the approved source set does not support the answer, ask for needed context where useful, or route the question to the responsible team. A plausible unsupported answer fails acceptance.

What must each RAG citation support?

No. The cited source must be accessible to the user and support the exact claim. The assistant also needs permission, freshness, conflict, and deletion tests because a citation can still point to stale or unauthorized material.

How should knowledge base permissions survive indexing?

The current user identity and source permission should constrain retrieval before content enters model context. Test role changes, tenant separation, shared links, caches, and permission revocation with paired users.

When is a limited RAG rollout acceptable?

Keep it limited when the assistant is useful for a bounded, low-consequence use but one risk still requires manual control, such as delayed freshness or an exception queue. Document the limit and the evidence needed to remove it.