Why your AI search or RAG returns the wrong documents

Evaluate retrieval separately from the answer

Build a test set that pairs real questions with the passages that should support the answer. Include questions with no valid document. Inspect the retrieved material before generating a response: a model cannot reliably repair a missing or irrelevant source. Also check whether the passage belongs to the right customer, product version and effective date. Relevance and permission are separate requirements.

What to check

• Inspect document parsing for missing headings, broken tables, repeated footers and lost page context.
• Check whether chunks keep necessary definitions, exceptions and dates together.
• Test access filters, stale documents and similarly worded material from different products or accounts.

What to change

Fix ingestion and metadata before adding more retrieval steps. Compare chunking strategies on the same questions. Consider keyword or hybrid retrieval when exact codes and names matter. If ranking adds value, measure it against a baseline. Cite the source passage in the answer and let the system decline when the evidence does not support a response.

How to evaluate the result

Track whether the required source appears in the retrieved set and whether the final response is supported by it. Report retrieval success separately from answer quality. Review false matches on documents that look similar but describe different policies.

What the result does not establish

Increasing the number of retrieved passages can add noise and cost. More context is useful only when it improves the relevant evidence available to the model without crossing access boundaries.