← Learn / Notebooks

Models & toolsPractitionerEvaluation notebook

Retrieval-Augmented Generation (RAG): Retrieval Evaluation

Evaluate ranked evidence with precision, recall and reciprocal rank, then trace whether an answer uses the context it received.

Before you start: Know what a ranked search result is; simple ratios. Python is optional.

AI-assisted study notebook · Published 14 Nov 2026. Source-support checks and synthetic examples are recorded; automated review is not independent expert fact verification.

In [1]: Architecture

1. RAG Evaluation Workflow

In retrieval-augmented generation (RAG), retrieved documents join the question as input to a generator. The saved Hugging Face source describes this basic arrangement. [1] A useful evaluation therefore asks two questions: did search return relevant material, and does the answer use its supplied context appropriately? Microsoft’s documentation and the Ragas paper distinguish these dimensions. [5] [6]

Workflow and scope notes
  1. Freeze the collection and query set. Keep relevance judgments for evaluation.
  2. Record ranked passage IDs, then calculate retrieval metrics against those labels.
  3. Record which passage text is actually supplied to the generator.
  4. Generate from the question and selected context.
  5. Check answer claims against supplied context and assess whether the answer addresses the question. Audit source reliability separately.
RAG evaluation: retrieval and supplied evidence
Evaluation sketch, not a universal RAG implementation. Judge retrieval against relevance labels; inspect context selection separately; compare answer claims with the exact supplied context. A relevant passage can be retrieved and still omitted before generation. Context support alone does not verify the source’s truth or currency. Source 1
Diagram as text and editable source
  • Question → Retriever: Query
  • Document collection → Retriever: Search
  • Retriever → Ranked passagesRecord IDs and ranks: Ranked results
  • Ranked passagesRecord IDs and ranks → Supplied contextRecord the exact text: Selection / packing
  • Supplied contextRecord the exact text → GeneratorQuestion + context: Evidence text
  • GeneratorQuestion + context → Answer claims: Generated answer
Download editable draw.io diagram
Retrieval and Support Boundaries

A relevant search hit is evidence available to the pipeline. It is not proof that the passage reached the generator, that the answer used it, or that the source itself is correct. The first two distinctions are visible in the trace below; source reliability needs a separate review.

In [2]: Interactive experiment

2. Interactive Retrieval Metrics

This is an invented six-passage manual for Demo Notes. Every passage has a declared binary label for each question. Rankings A and B are hand-authored examples, not outputs from real search systems. The experiment makes no claim about a particular model or retriever.

RANKING EXPLORERSynthetic · fully labelled

Cutoff k counts passages, not tokens. All six unique IDs are available; only the first k count toward these scores.

RANK / PASSAGERELEVANCE
  1. 01D1Choose Export in the notebook menu.Relevant
  2. 02D4The demo help desk opens from 09:00 to 17:00 UTC.Other
  3. 03D6Choose a theme under Appearance.OtherEND OF TOP 3 · 1 RELEVANT PASSAGE
  4. 04D2Select JSON as the output format.Relevant
  5. 05D5Deleted demo drafts stay in Trash for seven days.Other
  6. 06D3Download the file when its status says Ready.Relevant

Relevant for this question Labels are assigned, not model judgments.

Precision@30.3331 relevant / 3 retrieved
Recall@30.3331 found / 3 relevant
Reciprocal rank@31.000First relevant at rank 1

Ranking A · top 3 · found 1 of 3 relevant passages. D2 and D3 remain outside the cutoff.

TRY THIS

For the export question, move A from k = 3 to k = 4: precision rises from 1/3 to 2/4, and recall rises from 1/3 to 2/3. Then compare k = 4 with k = 5: the extra item is irrelevant, so precision falls to 2/5 while recall stays 2/3.

Observe: on one fixed ranking, adding passages cannot remove an already found relevant item. Precision can move in either direction. These are consequences of the formulas and this declared label set, not a claim that larger context improves answers.

In [3]: Definitions and assumptions

3. Precision, Recall and Reciprocal Rank

Let Tₖ be the first k unique returned IDs and R the complete relevant-ID set for this toy query. We apply binary precision and recall to that cutoff set. The underlying ratios follow the scikit-learn definitions; treating the first k results as the retrieved set is the lab’s explicit setup. [2] [3]

What share of this set is relevant?

Precision@k

|Tₖ ∩ R| / k

Relevant items divided by retrieved items. Here the retriever always returns at least k results.

Definition [2]
What share of the relevant set did we find?

Recall@k

|Tₖ ∩ R| / |R|

The denominator includes relevant items outside the cutoff. It is not the number retrieved.

Definition [3]
How soon did the first relevant item appear?

RR@k

1 / first relevant rank

Use zero if there is no relevant result in the first k. Later relevant items do not change this score.

Implementation [4]

Mean reciprocal rank (MRR@k) averages RR@k across all evaluated queries, including zero-scoring misses. BEIR’s pinned implementation uses the first positive relevance judgment and averages over its qrels query count. Our fixture provides rankings for every query. [4]

Declare the edge cases

This lab rejects duplicate IDs, rankings shorter than k, and queries with an empty relevant set. Those are teaching-code choices, not a universal metric standard. For a real evaluation, record its conventions. If judgments are incomplete, call the result recall against the labelled set; do not present it as complete real-world coverage.

Precision and recall at one cutoff use set membership. Swapping two items inside that set can leave both unchanged while changing reciprocal rank. RR also ignores whether a second or third required passage is present, so it cannot stand in for recall. These limitations follow directly from the definitions.

In [4]: Comparative view

4. Ranking Comparison

The comparison below gives each of the three toy questions equal weight. Mean precision and mean recall are arithmetic means of per-query values; MRR is mean RR. It is a tiny worked calculation, not a benchmark. Changing k in the explorer updates this figure and its table.

FIGURE 02Three queries · cutoff 3
Ranking ARanking BHigher score →

Wide chart: scroll horizontally; exact values are also in the table below.

Mean retrieval scores for the three fictional questionsAt cutoff three, Ranking A has mean precision 0.222, mean recall 0.444 and MRR 0.444. Ranking B has mean precision 0.556, mean recall 1.000 and MRR 0.833. Exact values are in the following table. 00.250.500.751.00 Mean precisionMean recallMRR 0.2220.556 0.4441.000 0.4440.833
Equal-weight query means at k = 3
RankingMean P@kMean R@kMRR@k
A2/9 = 0.2224/9 = 0.4444/9 = 0.444
B5/9 = 0.5561 = 1.0005/6 = 0.833
Original calculations from metrics_lab.py. The table is the accessible alternative to the chart. A source fact or software performance result is not being measured.

Per-query results. At k = 3, Ranking A misses the Trash passage completely, while Ranking B puts it first. That difference helps explain the aggregate. On this fixture B has better scores; it does not establish that a reranker or model will improve your own collection.

Suggested protocol for your own collection
  1. Freeze query IDs, corpus revision, relevance rules and a test set before comparing changes.
  2. Inspect per-query failures as well as aggregate scores; retain hard and unanswered questions.
  3. Change one retrieval setting at a time, then check both retrieval and answer behavior.
  4. Keep tuning data separate from a final evaluation set and record the actual inputs.

This is an editorial experiment plan, not a validated deployment recipe or a promised improvement.

In [5]: Failure diagnosis

5. Retrieval and Answer Failure Analysis

These hand-authored traces isolate different problems. The distinction between context relevance, answer faithfulness and answering the user’s actual question follows Ragas; the trace contents and diagnoses are this notebook’s examples. [6]

Context Selection Omission

QuestionHow do I export my notes?
RetrievedD1 · D2 · D3
Supplied contextD1 · D2 — D3 is absent
AnswerChoose Export, select JSON, then download when Ready.

Inspect context selection. D3 appears in retrieval, but the generator receives only D1 and D2. The download instruction is absent from that supplied context, even though it exists in the collection.

Here, “unsupported” means not supported by the supplied text. It is not a judgment that a statement is false in the world. Groundedness concerns consistency with context; evaluating the underlying sources is a different job. [5]

In [6]: Python · standard library only

6. Python Metric Calculation

The original Python lab uses exact rational arithmetic. It checks the three-query result and the fixed-ranking recall property across every permutation of the six IDs and every cutoff. The browser uses the same declared corpus and labels; its numerical results are checked against the Python output. The excerpt below uses the fixture declared in the full downloadable file.

metrics_lab.py · core calculation
from fractions import Fraction

# CASES is declared in the full downloadable lab.

def metrics(ranking, relevant, k):
    if not 1 <= k <= len(ranking):
        raise ValueError("Require 1 <= k <= number of returned results")
    if len(ranking) != len(set(ranking)):
        raise ValueError("Duplicate document IDs must be resolved first")
    if not relevant:
        raise ValueError("Recall needs a declared zero-relevance convention")
    top = ranking[:k]
    hits = sum(doc in relevant for doc in top)
    first = next((rank for rank, doc in enumerate(top, 1)
                  if doc in relevant), None)
    return {
        "precision": Fraction(hits, k),
        "recall": Fraction(hits, len(relevant)),
        "rr": Fraction(1, first) if first is not None else Fraction(0),
    }


def macro_scores(mode, k):
    rows = [metrics(case[mode], case["relevant"], k)
            for case in CASES.values()]
    return {name: sum(row[name] for row in rows) / len(rows)
            for name in rows[0]}

Download the full executable lab · Download exact test results

EXECUTED OUTPUT · FULL LAB
Permutation/cutoff checks: 4320
At k = 3:
  A: mean P = 2/9, mean R = 4/9, MRR = 4/9
  B: mean P = 5/9, mean R = 1,   MRR = 5/6

This verifies the arithmetic and fixture behavior, not retrieval quality on real data, scientific truth, or a deployed RAG system. See the review report for the recorded command and runtime.

In [7]: Knowledge check

7. Retrieval Evaluation Questions

Open a question and choose one answer. Grading reveals an explanation and its evidence. Questions refer to this notebook’s definitions and synthetic fixture.

Q1 · PrecisionFor export query A at k = 3, one of three returned passages is relevant. What is precision@3?
For export query A at k = 3, one of three returned passages is relevant. What is precision@3?
Answer and explanation

Answer: 1/3. Precision uses the retrieved set as its denominator: one relevant item divided by three returned items. [2]

Q2 · RecallThree passages are relevant in the collection, and one is in the cutoff. What is recall@3?
Three passages are relevant in the collection, and one is in the cutoff. What is recall@3?
Answer and explanation

Answer: 1/3. Recall divides one found relevant passage by the three relevant passages in the declared collection. [3]

Q3 · MRRAcross three queries, first-hit ranks inside k are 1, 3 and no hit. What is MRR@k?
Across three queries, first-hit ranks inside k are 1, 3 and no hit. What is MRR@k?
Answer and explanation

Answer: 4/9. The query scores are 1, 1/3 and 0. Their mean is (1 + 1/3 + 0) / 3 = 4/9. The miss stays in the denominator. [4]

Q4 · RankingTwo top-three lists contain the same two relevant IDs. What can change when their order changes?
Two top-three lists contain the same two relevant IDs. What can change when their order changes?
Answer and explanation

Answer: Reciprocal rank@3 can change. The top-three sets have the same membership, so precision and recall are unchanged. Moving the first relevant item changes RR. [4]

Q5 · ContextD3 was retrieved but removed from the supplied context. Where does this trace first lose that evidence?
D3 was retrieved but removed from the supplied context. Where does this trace first lose that evidence?
Answer and explanation

Answer: Context selection / packing. The trace shows D3 present in retrieval and absent in context. Inspect that boundary before attributing the problem to another stage. [5]

Q6 · SourcesA claim is supported by the supplied context. What does this establish by itself?
A claim is supported by the supplied context. What does this establish by itself?
Answer and explanation

Answer: Support relative to that context. Context support is a bounded relationship. Checking publisher reliability, dates and independent corroboration remains separate. [6]

In [8]: Evidence and limits

References and source links

  1. Kamsetty, A. (2021). Retrieval Augmented Generation with Huggingface Transformers and Ray. Anyscale guest article on Hugging Face.https://huggingface.co/blog/ray-rag
  2. scikit-learn maintainers. precision_score. Official scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html
  3. scikit-learn maintainers. recall_score. Official scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html
  4. BEIR maintainers. (2025 file commit). custom_metrics.py — mrr(). Commit 6d811867c38b01f1afc9b5a81b1dd609eb4f897c.https://raw.githubusercontent.com/beir-cellar/beir/6d811867c38b01f1afc9b5a81b1dd609eb4f897c/beir/retrieval/custom_metrics.py
  5. Microsoft. (Updated 2026). Retrieval-Augmented Generation (RAG) evaluators. Microsoft Learn.https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators?view=foundry
  6. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023; revised 2025). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217v2.https://arxiv.org/html/2309.15217v2
Source checks and provenance

Six primary references support the definitions and conceptual distinctions. One saved ResearchNode original was also checked against its public page. The metric implementation is pinned to a commit and the research paper to version 2. No private library material is included.

[1] Retrieval Augmented Generation with Huggingface Transformers and Ray

Accessed 13 Nov 2026. Basic retrieved-document plus generator arrangement. Evidence location: What is Retrieval Augmented Generation (RAG)?.

ResearchNode document 46f3ffc4bbd079e05a8ded5ad882e391
Revision e53ba23d35dfe41db4dce1ab971ee509

Limit: Dated architecture explanation; no old installation instructions or performance claims are reused.

[2] precision_score — scikit-learn 1.9.1 documentation

Accessed 13 Nov 2026. Binary precision ratio; the lab explicitly maps retrieved top-k items to predicted positives. Evidence location: API description, first definition paragraph.

Limit: A classification definition, not a prescribed RAG cutoff or deployment policy.

[3] recall_score — scikit-learn 1.9.1 documentation

Accessed 13 Nov 2026. Binary recall ratio, including relevant items that were not retrieved. Evidence location: API description, first definition paragraph.

Limit: The labelled set is complete only in this synthetic fixture.

[4] BEIR custom_metrics.py · mrr()

Accessed 13 Nov 2026. First positive relevance hit, zero for a miss inside k, and query-count averaging. Evidence location: mrr function, lines 21–30.

Limit: Source implementation inspected, never executed. Our fixture has a ranking for every judged query.

[5] Retrieval-Augmented Generation (RAG) evaluators

Accessed 13 Nov 2026. Separate document-retrieval evaluation from final-response groundedness and relevance. Evidence location: Process evaluation and System evaluation sections.

Limit: Product-specific evaluator scores are not used here. The public body was readable despite a generic authorization notice in the page template.

[6] Ragas: Automated Evaluation of Retrieval Augmented Generation

Accessed 13 Nov 2026. Distinguish faithfulness, answer relevance and context relevance. Evidence location: Section 3, Evaluation Strategies.

Limit: Full Section 3 read. No claim that its LLM judges independently verify facts or match current package behavior.

What this notebook establishes

Source passages were checked for support, calculations were reproduced, and the rendered layouts were inspected. This is an AI-assisted notebook with an automated source-support review, not independent expert fact verification. The example is small and fully labelled; it provides no real-system accuracy, latency or safety guarantee.