RAG Evaluation: Test Retrieval Before Tuning the LLM

RAG evaluation should start with retrieval. If the right evidence never reaches the model, prompt tuning and model upgrades only make the wrong context sound more convincing. Test retrieval, generation, and operations as separate stages. Keep a small, versioned regression set that fails when any stage gets worse.

Reader level: Intermediate. This guide assumes you know that retrieval-augmented generation, or RAG, finds documents before asking a language model to answer.

In this guide

RAG evaluation has three separate failure surfaces

A RAG application is a pipeline. One score for the final answer hides which component needs work.

The order matters. Retrieval is upstream. A generator cannot cite a policy paragraph that never entered its context, and a fluent answer does not prove that the retriever worked.

Microsoft's current RAG evaluator documentation makes the same separation in implementation terms. It provides document-retrieval measures for ranked evidence, then evaluates groundedness, relevance, and response completeness at the answer layer. The document-retrieval evaluator includes fidelity, NDCG, XDCG, maximum relevance, and missing relevance judgments.

_A useful RAG scorecard keeps retrieval, generation, and operational checks visible as separate layers._

Why one end-to-end score gives weak debugging evidence

Several research frameworks reach the same practical conclusion from different directions.

RAGChecker evaluates retriever and generator behavior separately. Its authors compared eight RAG systems across ten domains. Their metrics include claim recall and context precision for retrieval, plus context utilization, noise sensitivity, hallucination, and faithfulness for generation. In a 280-pair meta-evaluation, RAGChecker's overall-assessment score had a Spearman correlation of 0.609 with human preference. The two human annotators reached 0.689. Automated evaluation was useful, but it did not remove the human gap.

Ragas proposed reference-free measures for faithfulness, answer relevance, and context relevance. In its WikiEval comparisons, agreement with human preferences was 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance. The authors found context relevance hardest to judge. Treat those figures as results from that study, not universal accuracy rates for every judge, dataset, or domain.

A newer framework, RAGe, adds component selection and hardware telemetry. It evaluates pipeline configurations across chunking, embedding, retrieval, storage, and generation while pruning combinations that exceed latency or VRAM limits. The paper uses Natural Questions, NewsQA, and TriviaQA by default and supports custom CSV or JSON datasets. Its main contribution is a way to compare quality with resource constraints; it does not establish one configuration that wins across domains.

Together, these papers support a diagnostic approach. They do not prove that a particular metric or library is sufficient for production.

Build a test set before choosing metrics

Start with 40 to 60 questions from the domain you serve. That range is a practical starting point rather than a statistical law. It is large enough to expose several failure types and small enough for a human to review after every material change.

Include at least five query classes:

Production logs can suggest questions, but remove personal data and secrets before adding examples to an evaluation set. A recent practitioner discussion about production RAG evaluation also emphasizes fixed queries, versioned configurations, and separate retrieval and generation checks. That discussion is anecdotal evidence about workflow pain, not proof that the approach works in every system.

Store judgments against stable document IDs alongside any copied text. Chunk boundaries change when you adjust a splitter. A canonical source ID lets the same test survive that change.

{
  "query_id": "refund-window-01",
  "query": "How long does a customer have to return an unopened item?",
  "relevant_document_ids": ["returns-policy-v4"],
  "required_claims": ["Unopened items may be returned within 30 days."],
  "must_abstain": false,
  "allowed_roles": ["customer", "support"],
  "as_of": "2026-07-01"
}

For an unanswerable case, set `relevant_document_ids` to an empty list and `must_abstain` to `true`. For a restricted case, run the same query under two roles. The authorized user should retrieve the document; the unauthorized user should not learn that document's contents.

Teams building their own corpus and application layer should version the content contract alongside the test set. The same rule applies to custom data systems built with Next.js and AI: a schema or content change can alter retrieval without touching the prompt.

Step 1: evaluate retrieval without generating an answer

Run each query through the retriever and save the ranked result IDs, scores, timestamps, and access decisions. Do not call the language model yet.

Pick metrics that match the evidence shape

Use hit rate@k when one correct document is enough. It asks whether at least one relevant source appears in the first `k` results.

Use recall@k when the answer requires several sources. It measures how many known relevant documents appeared in the first `k`.

Use precision@k when irrelevant context is expensive or distracting. High recall with low precision can flood the generator with noise.

Use MRR when the first relevant result matters most. Use NDCG when several graded results should appear in a useful order. Microsoft documents NDCG and related ranked-retrieval measures in its evaluator, while RAGChecker uses claim recall and context precision to connect retrieved evidence to the claims an answer needs.

Do not collect every metric by default. Pick one coverage measure and one ranking or noise measure. Add a metric only when it changes a decision.

Classify misses before changing the model

Retrieval failures usually fall into a small set:

Each failure has a different owner. Re-embedding cannot repair a missing document. A bigger language model cannot repair a permissions filter. A reranker may help when the evidence is present but badly ordered.

For tool-connected applications, preserve the retrieval request, filters, result IDs, and tool response in the trace. That fits the broader control pattern described in MCP developer workflows: inspect the contract, state transition, and final prose.

Step 2: evaluate generation over fixed evidence

Once retrieval meets its threshold, freeze the retrieved contexts and replay them against the generator. This isolates prompt or model changes from index changes.

Measure four behaviors:

A reference answer can help with completeness. It is less useful as the only truth source because several wordings may be correct. Store required claims and supporting document IDs when possible.

Run a second generation test with deliberately incomplete context. A reliable system should expose uncertainty rather than fill gaps from model memory. This matters when the corpus contains private, changing, or domain-specific facts.

Model choice still affects answer quality, latency, and cost, but it comes after retrieval evidence. If the same fixed context fails across generators, compare model and API trade-offs. If the context itself is wrong, changing the generator is wasted motion.

Add security and conflict cases to the retrieval suite

Ordinary relevance tests miss adversarial or conflicting evidence.

A July 2026 paper on polymorphic sybil poisoning in RAG tested groups of lexically different passages that supported the same attacker-selected answer. Under the paper's forced-exposure setup, polymorphic passages produced a 22.8% hijack rate compared with 4.0% for repeated monomorphic passages. Token-overlap filtering caught all monomorphic clusters and none of the polymorphic clusters.

The result does not measure how often this attack succeeds in production. The authors fixed the retrieved mix at six attack passages, two gold passages, and two fillers to isolate reader behavior. They also report limitations around one attack class, dataset contamination risk, a 500-question ablation, and LLM-based verification.

The useful evaluation lesson is narrower: classify more than “correct” and “attacker target.” The paper tracks four outcomes:

Add conflict cases to your own set. Include duplicated claims with different wording, an outdated source that contradicts the current policy, and a lower-trust source that conflicts with an authoritative one. Record whether the system answers, abstains, or drifts.

Calibrate LLM judges before trusting their scores

LLM judges make regression testing cheaper, especially for groundedness and claim coverage. They remain software dependencies with prompts, model versions, parsing behavior, and known blind spots.

Use four controls:

The Ragas and RAGChecker studies both show why calibration matters. Agreement varies by dimension, and automated correlations remain below human agreement. A numeric score should trigger inspection, not end it.

Current open-source releases also show active work around evaluators. DeepEval 4.1.3, released July 12, 2026, added deterministic checks for agent loops and tool permissions while fixing Ragas integration. TruLens 2.9.0, released July 23, added judge ensembles, A/B criteria tests, score-distribution analysis, and golden-set generation. Release activity is evidence of maintained engineering work, not evidence that either library is the right choice for your stack.

A minimal RAG regression workflow

Use the same sequence for every material pipeline change:

A compact result record can look like this:

{
  "run_id": "rag-2026-07-26-b",
  "test_set": "support-v7",
  "corpus_snapshot": "2026-07-25T22:00:00Z",
  "retrieval": {
    "recall_at_5": 0.91,
    "ndcg_at_5": 0.84,
    "unauthorized_hits": 0
  },
  "generation": {
    "grounded_pass_rate": 0.94,
    "complete_pass_rate": 0.87,
    "abstention_pass_rate": 0.90
  },
  "operations": {
    "p95_ms": 1380,
    "cost_per_query_usd": 0.0042
  }
}

Those numbers are illustrative. Set thresholds from your risk, baseline, and error cost. A medical knowledge assistant and a product-search helper should not share the same release gate.

Decide the fix from the failed layer

This table is the point of RAG evaluation: a failed score should identify the next experiment. If it cannot, the metric is too far from the component you need to change.

What the evidence supports

The papers measured specific systems and datasets. RAGChecker found that modular metrics can correlate with human preferences and expose retriever-generator trade-offs. Ragas found that judge agreement varied across faithfulness, answer relevance, and context relevance. RAGe demonstrated a framework that combines quality metrics with latency and memory constraints. The poisoning benchmark showed that one constrained attack setup produced distinct hijack, abstention, and drift patterns.

The evidence does not establish universal thresholds, a universally best evaluator, or production attack prevalence. My practical interpretation is to separate the stages, keep a human-calibrated slice, and require each metric to point to an engineering action.

Claim checks

Sources