An employee asks a company assistant: “What is the travel reimbursement limit for an overseas trip?”
The system returns a clear answer. It cites a policy. The response looks reliable.
There is one problem. The cited policy was withdrawn six months ago.
This is where many enterprise retrieval-augmented generation (RAG) pilots meet production reality. The model understood the question. Retrieval found a highly relevant passage. The citation was valid. The answer was still wrong because the system treated semantic relevance as business authority.
The executive question is therefore not whether RAG can answer questions. It is whether the organisation can trust those answers in real work.
This article sets out four tests for production RAG: relevance, authority, evidence and operability. It also shows where controls need to sit before retrieval, during evidence selection and before an answer is released.
Enterprise RAG reliability is a system property. It cannot be created by the model or prompt alone.
At a glance
- Retrieving a related document is not the same as retrieving valid evidence.
- A vector index helps find content. It does not determine which source is approved, current or applicable.
- Material answers should be traceable to specific evidence, not only to a document link.
- A production system must know when to refuse, escalate and operate within access boundaries.
Four tests separate a convincing demo from a reliable service
Every material answer should pass four tests.
Relevance: did the system retrieve the right information?
The system must find the applicable clause, table, appendix or business context—not merely content that resembles the question.
A relevance failure occurs when the retrieved passage sounds appropriate but does not address the user’s exact circumstances.
Authority: is the information approved and applicable?
A source can be relevant without being authoritative.
The system must determine whether the document is approved, current, applicable to the user’s geography and permitted for the intended purpose.
An obsolete, draft or wrong-region document should not outrank the current source simply because its language resembles the question more closely.
Evidence: can the answer be verified?
A material claim should be traceable to the specific passage, rule and document version that support it.
Linking to a long document is not enough if the reviewer must search through it again to verify the answer.
Operability: can the organisation run the service safely?
The system must remain reliable as documents, permissions, policies, models and platforms change.
It must also provide enough operational evidence to investigate incidents, correct failures and understand how an answer was produced.
The travel-policy example shows why relevance is only the first step
In a demonstration, the knowledge set is usually small and clean. Production repositories are not.
They contain duplicates, withdrawn policies, regional variants, scanned documents, attachments and inconsistent metadata.
For the travel question, semantic search may retrieve the withdrawn policy because its wording closely matches the employee’s query. A reliable design must also establish which policy is effective, whether a local rule overrides a global one and whether the employee is entitled to see the source.
The same issue appears in multilingual environments. A question may arrive in Arabic while the authoritative policy is in English. Retrieval still has to preserve the original meaning and trace the response back to the controlled source.
The practical implication is straightforward: retrieval quality should be tested against enterprise ambiguity, not only against curated demonstration questions.
Authority has to be engineered explicitly
A vector database is an index. It does not know that one policy was withdrawn, another is still a draft or a regional policy took effect last week.
That logic has to come from the application around the model.
Before a source can be used as evidence, the system may need to check:
- Document version
- Approval status
- Effective date
- Applicable geography
- Document precedence
- User access rights
Some decisions should remain deterministic. A model may interpret a clause. It should not invent the organisation’s document hierarchy or decide that a draft overrides an approved policy.
The system also needs an abstain path. When authoritative evidence is missing or conflicting, “the available sources do not support a reliable answer” can be the correct outcome.
Evidence must be specific enough to review
A citation is useful only if a reviewer can verify the answer without repeating the search. A link to a 200-page manual is not enough for a material decision.
Depending on the workflow, the answer may need to retain:
- The source and its version
- The page, clause or section used
- The specific passages supporting the answer
- The business rules applied
- Any conflicting evidence found
For higher-risk use cases, unsupported recommendations may need to be blocked or routed for approval.
This traceability also improves debugging. When a user rejects an answer, the team can determine whether the failure came from the source data, retrieval process, authority rules, translation, model or surrounding workflow.
Without that evidence, teams often change prompts without knowing what actually failed.
Controls belong across the pipeline, not around the edges
The control model can remain compact. Three points matter most.
Before retrieval
The system should validate the request, establish the user’s identity, apply access rights and route the question to the correct business domain.
These controls reduce the risk of processing invalid, prohibited or misdirected requests.
During evidence selection
The system should check version, approval status, effective date, geography, document precedence and user entitlement.
These controls prevent relevant but obsolete, unauthorised or inapplicable information from being treated as evidence.
Before answer release
The system should validate material claims against their sources, confirm citations, apply disclosure rules and escalate when reliable evidence is unavailable.
These controls reduce unsupported, unsafe or unverifiable responses.
These are not optional filters added after the model is built. They are part of the application’s business and control architecture.
Production also changes the ownership problem
A pilot can operate with fixed documents and a temporary project team. A production service cannot.
Someone has to own:
- Source quality
- Obsolete and duplicate content
- Access synchronisation
- Answer policies
- Incident handling
- Model and platform changes
Session state also needs care. Information from an earlier conversation should not become trusted evidence simply because the model remembers it.
The architecture should preserve the ability to change. Retrieval, rules, workflow controls and integration logic should not be tightly coupled to one model or hosting platform.
Public cloud may fit some environments. Others may require private, sovereign or on-premises deployment.
For on-premises RAG, reliability does not require an immediate commitment to a large GPU estate. Start with proven demand and minimum viable capacity. Expand when workload and utilisation justify it.
Moving from pilot to production changes the test
A successful pilot proves that the idea can work under selected conditions. Production must prove that it can continue working under normal enterprise conditions.
Relevance
A pilot may search a carefully selected document set.
Production must retrieve correctly across formats, languages, metadata inconsistencies and access boundaries.
Authority
A pilot may assume that every supplied document is valid.
Production must apply approval, version, precedence and effective-date rules before using a source.
Evidence
A pilot may display a link to the retrieved document.
Production must trace material claims to specific passages, rules and source versions.
Operability
A pilot may run with fixed data and support from the project team.
Production must be monitored, secured, supported and changed through normal operational processes.
The step-up is therefore not cosmetic. It changes the engineering, control and ownership model around the system.
What leaders should require before scaling RAG
Define the decision boundary
State what the system may answer, recommend or initiate.
An employee policy assistant has a different risk profile from a system that recommends compliance, contractual or financial actions.
Approve the source-of-truth model
Identify the authoritative repositories, document precedence, approval states and effective-date rules.
This should be an explicit business and control decision—not something left for developers to infer.
Test failure conditions
Test obsolete documents, conflicting policies, missing evidence, unauthorised sources, multilingual questions and cases that should not be answered.
A production test set should include ambiguity and failure, not only examples where the correct answer is available.
Assign operational ownership
Before rollout, name the owners of source quality, access control, answer policy, operations and incident handling.
Without clear ownership, reliability will deteriorate as the underlying information changes.
Reliability is engineered, not prompted
Enterprise RAG should not be judged by how convincing its best answer looks.
It should be judged by whether the organisation can explain why the answer was produced, whether the right sources were used, whether the service stayed within its authority and whether it can continue operating as data, policies and models change.
A better model can improve language quality. A better prompt can reduce some errors. Neither replaces sound engineering.
A reliable RAG system knows what to retrieve, what to trust, how to prove its answer—and when not to answer at all.
Are you moving a RAG workflow from pilot to production?
Bring two or three workflows, the repositories they depend on and the decisions the system is expected to support.
Ampersand can help determine where retrieval is sufficient, where authority and deterministic controls are required, what evidence should be retained and what needs to be engineered for safe operation.
Practical output: A focused production-readiness design for the selected RAG workflows.
