1. RAG Evaluation Workflow
In retrieval-augmented generation (RAG), retrieved documents join the question as input to a generator. The saved Hugging Face source describes this basic arrangement. [1] A useful evaluation therefore asks two questions: did search return relevant material, and does the answer use its supplied context appropriately? Microsoft’s documentation and the Ragas paper distinguish these dimensions. [5] [6]
Workflow and scope notes
- Freeze the collection and query set. Keep relevance judgments for evaluation.
- Record ranked passage IDs, then calculate retrieval metrics against those labels.
- Record which passage text is actually supplied to the generator.
- Generate from the question and selected context.
- Check answer claims against supplied context and assess whether the answer addresses the question. Audit source reliability separately.
Diagram as text and editable source
- Question → Retriever: Query
- Document collection → Retriever: Search
- Retriever → Ranked passagesRecord IDs and ranks: Ranked results
- Ranked passagesRecord IDs and ranks → Supplied contextRecord the exact text: Selection / packing
- Supplied contextRecord the exact text → GeneratorQuestion + context: Evidence text
- GeneratorQuestion + context → Answer claims: Generated answer
A relevant search hit is evidence available to the pipeline. It is not proof that the passage reached the generator, that the answer used it, or that the source itself is correct. The first two distinctions are visible in the trace below; source reliability needs a separate review.
2. Interactive Retrieval Metrics
This is an invented six-passage manual for Demo Notes. Every passage has a declared binary label for each question. Rankings A and B are hand-authored examples, not outputs from real search systems. The experiment makes no claim about a particular model or retriever.
Cutoff k counts passages, not tokens. All six unique IDs are available; only the first k count toward these scores.
- 01D1Choose Export in the notebook menu.Relevant
- 02D4The demo help desk opens from 09:00 to 17:00 UTC.Other
- 03D6Choose a theme under Appearance.OtherEND OF TOP 3 · 1 RELEVANT PASSAGE
- 04D2Select JSON as the output format.Relevant
- 05D5Deleted demo drafts stay in Trash for seven days.Other
- 06D3Download the file when its status says Ready.Relevant
Relevant for this question Labels are assigned, not model judgments.
Ranking A · top 3 · found 1 of 3 relevant passages. D2 and D3 remain outside the cutoff.
For the export question, move A from k = 3 to k = 4: precision rises from 1/3 to 2/4, and recall rises from 1/3 to 2/3. Then compare k = 4 with k = 5: the extra item is irrelevant, so precision falls to 2/5 while recall stays 2/3.
Observe: on one fixed ranking, adding passages cannot remove an already found relevant item. Precision can move in either direction. These are consequences of the formulas and this declared label set, not a claim that larger context improves answers.
3. Precision, Recall and Reciprocal Rank
Let Tₖ be the first k unique returned IDs and R the complete relevant-ID set for this toy query. We apply binary precision and recall to that cutoff set. The underlying ratios follow the scikit-learn definitions; treating the first k results as the retrieved set is the lab’s explicit setup. [2] [3]
Precision@k
|Tₖ ∩ R| / k
Relevant items divided by retrieved items. Here the retriever always returns at least k results.
Definition [2]Recall@k
|Tₖ ∩ R| / |R|
The denominator includes relevant items outside the cutoff. It is not the number retrieved.
Definition [3]RR@k
1 / first relevant rank
Use zero if there is no relevant result in the first k. Later relevant items do not change this score.
Implementation [4]Mean reciprocal rank (MRR@k) averages RR@k across all evaluated queries, including zero-scoring misses. BEIR’s pinned implementation uses the first positive relevance judgment and averages over its qrels query count. Our fixture provides rankings for every query. [4]
This lab rejects duplicate IDs, rankings shorter than k, and queries with an empty relevant set. Those are teaching-code choices, not a universal metric standard. For a real evaluation, record its conventions. If judgments are incomplete, call the result recall against the labelled set; do not present it as complete real-world coverage.
Precision and recall at one cutoff use set membership. Swapping two items inside that set can leave both unchanged while changing reciprocal rank. RR also ignores whether a second or third required passage is present, so it cannot stand in for recall. These limitations follow directly from the definitions.
4. Ranking Comparison
The comparison below gives each of the three toy questions equal weight. Mean precision and mean recall are arithmetic means of per-query values; MRR is mean RR. It is a tiny worked calculation, not a benchmark. Changing k in the explorer updates this figure and its table.
Wide chart: scroll horizontally; exact values are also in the table below.
| Ranking | Mean P@k | Mean R@k | MRR@k |
|---|---|---|---|
| A | 2/9 = 0.222 | 4/9 = 0.444 | 4/9 = 0.444 |
| B | 5/9 = 0.556 | 1 = 1.000 | 5/6 = 0.833 |
Per-query results. At k = 3, Ranking A misses the Trash passage completely, while Ranking B puts it first. That difference helps explain the aggregate. On this fixture B has better scores; it does not establish that a reranker or model will improve your own collection.
Suggested protocol for your own collection
- Freeze query IDs, corpus revision, relevance rules and a test set before comparing changes.
- Inspect per-query failures as well as aggregate scores; retain hard and unanswered questions.
- Change one retrieval setting at a time, then check both retrieval and answer behavior.
- Keep tuning data separate from a final evaluation set and record the actual inputs.
This is an editorial experiment plan, not a validated deployment recipe or a promised improvement.
5. Retrieval and Answer Failure Analysis
These hand-authored traces isolate different problems. The distinction between context relevance, answer faithfulness and answering the user’s actual question follows Ragas; the trace contents and diagnoses are this notebook’s examples. [6]
Context Selection Omission
Inspect context selection. D3 appears in retrieval, but the generator receives only D1 and D2. The download instruction is absent from that supplied context, even though it exists in the collection.
Here, “unsupported” means not supported by the supplied text. It is not a judgment that a statement is false in the world. Groundedness concerns consistency with context; evaluating the underlying sources is a different job. [5]
6. Python Metric Calculation
The original Python lab uses exact rational arithmetic. It checks the three-query result and the fixed-ranking recall property across every permutation of the six IDs and every cutoff. The browser uses the same declared corpus and labels; its numerical results are checked against the Python output. The excerpt below uses the fixture declared in the full downloadable file.
from fractions import Fraction
# CASES is declared in the full downloadable lab.
def metrics(ranking, relevant, k):
if not 1 <= k <= len(ranking):
raise ValueError("Require 1 <= k <= number of returned results")
if len(ranking) != len(set(ranking)):
raise ValueError("Duplicate document IDs must be resolved first")
if not relevant:
raise ValueError("Recall needs a declared zero-relevance convention")
top = ranking[:k]
hits = sum(doc in relevant for doc in top)
first = next((rank for rank, doc in enumerate(top, 1)
if doc in relevant), None)
return {
"precision": Fraction(hits, k),
"recall": Fraction(hits, len(relevant)),
"rr": Fraction(1, first) if first is not None else Fraction(0),
}
def macro_scores(mode, k):
rows = [metrics(case[mode], case["relevant"], k)
for case in CASES.values()]
return {name: sum(row[name] for row in rows) / len(rows)
for name in rows[0]}Download the full executable lab · Download exact test results
Permutation/cutoff checks: 4320 At k = 3: A: mean P = 2/9, mean R = 4/9, MRR = 4/9 B: mean P = 5/9, mean R = 1, MRR = 5/6
This verifies the arithmetic and fixture behavior, not retrieval quality on real data, scientific truth, or a deployed RAG system. See the review report for the recorded command and runtime.
7. Retrieval Evaluation Questions
Open a question and choose one answer. Grading reveals an explanation and its evidence. Questions refer to this notebook’s definitions and synthetic fixture.
References and source links
- Kamsetty, A. (2021). Retrieval Augmented Generation with Huggingface Transformers and Ray. Anyscale guest article on Hugging Face.https://huggingface.co/blog/ray-rag
- scikit-learn maintainers. precision_score. Official scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html
- scikit-learn maintainers. recall_score. Official scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html
- BEIR maintainers. (2025 file commit). custom_metrics.py — mrr(). Commit 6d811867c38b01f1afc9b5a81b1dd609eb4f897c.https://raw.githubusercontent.com/beir-cellar/beir/6d811867c38b01f1afc9b5a81b1dd609eb4f897c/beir/retrieval/custom_metrics.py
- Microsoft. (Updated 2026). Retrieval-Augmented Generation (RAG) evaluators. Microsoft Learn.https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators?view=foundry
- Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023; revised 2025). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217v2.https://arxiv.org/html/2309.15217v2
Source checks and provenance
Six primary references support the definitions and conceptual distinctions. One saved ResearchNode original was also checked against its public page. The metric implementation is pinned to a commit and the research paper to version 2. No private library material is included.
Accessed 13 Nov 2026. Basic retrieved-document plus generator arrangement. Evidence location: What is Retrieval Augmented Generation (RAG)?.
ResearchNode document 46f3ffc4bbd079e05a8ded5ad882e391
Revision e53ba23d35dfe41db4dce1ab971ee509
Limit: Dated architecture explanation; no old installation instructions or performance claims are reused.
Accessed 13 Nov 2026. Binary precision ratio; the lab explicitly maps retrieved top-k items to predicted positives. Evidence location: API description, first definition paragraph.
Limit: A classification definition, not a prescribed RAG cutoff or deployment policy.
Accessed 13 Nov 2026. Binary recall ratio, including relevant items that were not retrieved. Evidence location: API description, first definition paragraph.
Limit: The labelled set is complete only in this synthetic fixture.
Accessed 13 Nov 2026. First positive relevance hit, zero for a miss inside k, and query-count averaging. Evidence location: mrr function, lines 21–30.
Limit: Source implementation inspected, never executed. Our fixture has a ranking for every judged query.
Accessed 13 Nov 2026. Separate document-retrieval evaluation from final-response groundedness and relevance. Evidence location: Process evaluation and System evaluation sections.
Limit: Product-specific evaluator scores are not used here. The public body was readable despite a generic authorization notice in the page template.
Accessed 13 Nov 2026. Distinguish faithfulness, answer relevance and context relevance. Evidence location: Section 3, Evaluation Strategies.
Limit: Full Section 3 read. No claim that its LLM judges independently verify facts or match current package behavior.
Source passages were checked for support, calculations were reproduced, and the rendered layouts were inspected. This is an AI-assisted notebook with an automated source-support review, not independent expert fact verification. The example is small and fully labelled; it provides no real-system accuracy, latency or safety guarantee.