Note 01 / RAG · Evaluation
Evaluate RAG: separate retrieval from answers
Recall, hit rate and answer correctness measure different things. A small example and a reproducible verification plan.
A plausible answer is not evidence
When a RAG system answers incorrectly, two separate stages may be responsible: the retriever fails to find relevant text, or the generator misuses text it found. A single overall score hides the distinction. Check retrieved sources first, then the answer.
An evaluation case contains a question, relevant document or passage IDs, and criteria for an acceptable answer. Granularity and ID versions must remain stable. Changed chunking can otherwise appear to cause a regression when only the evaluation reference has changed.
Do not confuse recall and hit rate
Recall@k is the fraction of relevant items appearing among the first k unique results. Hit rate@k only indicates whether at least one relevant item appears. Reciprocal rank considers the position of the first relevant result. With several relevant documents, hit rate can look good while recall remains low.
def retrieval_metrics(expected, ranked, k):
if not expected or k < 1:
raise ValueError("non-empty expected set and positive k required")
# Deduplicate document IDs before calculating metrics.
top = list(dict.fromkeys(ranked))[:k]
hits = len(set(top) & expected)
first = next((rank for rank, doc in enumerate(top, 1)
if doc in expected), None)
return {
"recall": hits / len(expected),
"hit_rate": float(hits > 0),
"reciprocal_rank": 1 / first if first else 0.0,
}
# Synthetic functional test, not a production benchmark.
expected = {"doc-a", "doc-b"}
ranked = ["doc-x", "doc-a", "doc-a", "doc-b"]
result = retrieval_metrics(expected, ranked, k=2)
assert result == {"recall": 0.5, "hit_rate": 1.0,
"reciprocal_rank": 0.5}
The numbers in the assertion are expected results for a synthetic fixture. They do not describe the quality of a served RAG system. Across a dataset, calculate these values for each answerable question and then average them, producing mean recall and Mean Reciprocal Rank.
Answer validation and regression
Questions without a supported answer belong in a separate set. An empty relevance set is not zero recall; it is a different case testing whether the system recognises missing evidence. For answerable questions, also check whether cited passages support each substantive statement and whether the response addresses the question.
Version questions, corpus snapshot, embedding model, chunking and generator configuration alongside results. Define acceptance criteria before comparing changes. This note’s local Python test checks duplicate hits, missing hits and invalid inputs; it does not start an LLM or provide a production benchmark.
Limits of the test set
A small set cannot reliably cover rare failures. Add failed cases, retain a separate holdout set, and sample unchanged answers for review too. An LLM used as a judge needs calibration and human cross-checking. For latency, document hardware, load, input lengths, measurement boundaries and repetitions; this note publishes no such measurements.