Guide / RAG retrieval problems

RAG Retrieval Failure Modes

Most bad RAG answers start before generation. Debug the retrieved evidence before blaming the model.

The failure usually starts in retrieval

If the retriever returns the wrong evidence, the generator can still produce fluent text. Fluency hides retrieval failure unless the system logs candidates, scores, filters, and citations.

RAG quality should be inspected at the context boundary: what chunks were passed to the model, why were they selected, and what relevant chunks were missed?

Failure taxonomy

Common failures include missing exact identifiers, ambiguous queries, stale chunks, bad chunk boundaries, metadata filter errors, access-control leakage, and reranker overfitting.

Each failure has a different fix. Increasing top-k does not repair stale metadata. Changing embeddings does not repair missing permission filters.

A practical debugging loop

Run vector-only, BM25-only, and hybrid retrieval for the same query. Print top candidates with scores and metadata. Compare them to judged relevant chunks. Only then change chunking, filters, or index settings.

Complexity Table

Failure	Symptom	Likely cause	Fix
Wrong chunk	Answer cites related but irrelevant text	Vector-only semantic drift	Hybrid retrieval and reranking
Missing chunk	No answer despite source existing	Chunking or filters	Boundary repair and metadata audit
Stale context	Answer uses old policy/version	Weak freshness metadata	Version filters and reindexing
Permission leak	User sees restricted source	Late or missing ACL filter	Pre-filtered access control

When to Use This

Use this checklist when answer quality drops, citations look plausible but wrong, or a new corpus produces unexpected retrieval misses.
Use it before changing the LLM. Retrieval evidence is cheaper and more deterministic to inspect.

When Not to Use This

Do not use a retrieval checklist as a substitute for judged queries and human relevance labels.
Do not debug only the final answer when the retrieved context is available to inspect.

Production Failure Modes

The worst production failure is a confident answer grounded in the wrong retrieved chunk. It passes a superficial fluency check but fails the user task.

The second common failure is silent recall loss after re-embedding, changing chunking, or adding filters without rerunning the relevance test set.

Animated SVG Diagram

A RAG pipeline shows failure points at chunking, embedding, filtering, retrieval, reranking, and generation.

Next Topics

RAG chunking strategies production RAG retrieval architecture vector search limitations vector database memory calculator

FAQ

Why does my RAG system retrieve the wrong documents?

Common causes include ambiguous queries, poor chunking, missing metadata filters, vector-only retrieval, stale embeddings, and weak reranking.

How do I measure RAG retrieval quality?

Use judged queries and measure recall@k, MRR, NDCG, citation correctness, answer sufficiency, and latency by retrieval stage.

Should I increase top-k to fix RAG?

Sometimes, but it can add noise and reranker cost. First identify whether the relevant chunk is missing, filtered out, stale, or ranked too low.