Retrieval-augmented generation (RAG) connects a model to a document collection, but that connection does not automatically make its answers reliable. Common RAG failure modes include stale source versions, missing permissions, exceptions lost during retrieval and answers that misrepresent cited passages. Diagnose source preparation, search and answer generation separately: each failure requires a different fix.
RAG failures in the source collection
Identify authoritative documents and remove or label superseded material. Preserve effective dates, ownership and access rules. Conflicting versions can produce a well-cited answer that is still wrong for the current process. Source maintenance is an operating responsibility, not a one-time ingestion task.
Inspect what search returns
Review the passages retrieved for representative questions. Important conditions may sit in a table, an appendix or an adjacent section. Preparing passages without context can separate a rule from its exception. Anthropic’s contextual retrieval work addresses this context problem; any chosen method still needs evaluation on the organization’s documents and questions. Contextual retrieval research.
Check claim-to-source support
A source link can point to a real document without supporting the statement beside it. Test whether each material claim follows from the cited passage. Include questions that lack an answer in the collection. The system should expose missing or conflicting evidence rather than turning it into a confident synthesis.
Trace a wrong answer through the pipeline
Begin with the exact question and user permissions. Inspect the eligible source collection, then the retrieved passages, then the answer. If the correct passage was never available, prompt tuning will not solve the problem. If it was retrieved but the answer ignored an exception, the failure belongs later in the pipeline.
Use that trace to create a targeted regression case. Include the expected source, the relevant qualification and the acceptable response when evidence is absent. Keep the original failure alongside the corrected behavior. This makes future changes easier to evaluate and prevents one local fix from hiding a different weakness elsewhere.
Build a diagnostic acceptance set
Label failures by source, permission, retrieval or synthesis so the team can fix the right layer. Add corrected failures to regression tests. After a document update, verify both that the new content is found and that the retired version is no longer used incorrectly. Track reviewer corrections as a quality signal in operation.
Before the next decision
Planning checklist
- Authoritative collection and version lifecycle
- Permission tests before search
- Retrieval checks for exceptions and tables
- Claim-level citation and no-answer tests
Sources and further reading
Sources checked 5 October 2026. The design recommendations in this guide are AID’s analysis. Sector obligations depend on the use case and jurisdiction.
Apply this to a defined workflow
AID’s Regulated RAG Accelerator can address source preparation, controlled retrieval and evaluation as one implementation scope.
