← Learn / Notebooks

Models & toolsPractitionerStudy guide

Model Evaluation: Test Sets and Data Leakage

Compare row splits with held-out groups and see how a memorizing baseline can produce a misleading result.

Before you start: Training and test sets, binary labels and simple accuracy.

AI-assisted study notebook · Published 14 Nov 2026. Source-support checks and synthetic examples are recorded; automated review is not independent expert fact verification.

In [1]: Architecture

1. Architecture and scope

An evaluation split encodes a prediction question. If the intended use is a new group, testing on another row from a training group can answer an easier question. scikit-learn documents group-aware splitting to keep a group out of both training and test simultaneously. The relevant unit might be a person, source document or other correlated group in an actual task. [3]

Data leakage uses information during model building that would not be available at prediction time. Test results should not drive model choices, and fitted preprocessing belongs on the training data. The saved forecasting tutorial also makes its train, validation and test windows explicit; its temporal design is context rather than a universal split recipe. [1] [2]

Scope. A synthetic grouped dataset and a memorizing rule. No real model is trained and no production accuracy is estimated.

Workflow and scope notes
  1. Define prediction. State what will be new when the system is actually used. This example targets groups absent from training.
  2. Choose split unit. Separate data using the unit appropriate to that prediction question. Keeping unique row IDs apart does not guarantee groups are separate.
  3. Fit on training. Learn or choose behavior from training data and designated validation work. The toy memorizes group labels and defaults to 0 for unseen groups.
  4. Evaluate once. Compute the stated metric on the held-out data and report the split design. Repeatedly choosing from test results makes those results part of model selection.
In [2]: Experiment

2. Row split versus held-out groups

Groups A, B, C and D each have two records. Their fixed labels are 0, 1, 0 and 1. The rule memorizes a group’s training label and predicts 0 for an unseen group. Choose the split and inspect overlap alongside accuracy.

Row split versus held-out groups

Original synthetic experiment · Change an input to update the figure.

Row split: A1 B1 C1 D1 train; A2 B2 C2 D2 test. Group split: all A/B train; all C/D test. The label rule is deliberately constructed. Its 100% and 50% scores do not estimate the effect size of leakage in any real dataset.

Split membership and synthetic accuracy4 shared groups · Accuracy 100% · Predictions 0, 1, 0, 1. Both accuracy bars describe the fixed memorizing rule, not a trained model.GROUP MEMBERSHIPABCDTrainPresentPresentPresentPresentTestPresentPresentPresentPresentShared groups: A, B, C, DRow split100%Held-out groups50%
4 shared groups · Accuracy 100% · Predictions 0, 1, 0, 1
Calculation record
{
  "inputs": {
    "split": "row"
  },
  "train": [
    "A",
    "B",
    "C",
    "D"
  ],
  "test": [
    "A",
    "B",
    "C",
    "D"
  ],
  "overlap": [
    "A",
    "B",
    "C",
    "D"
  ],
  "predictions": [
    0,
    1,
    0,
    1
  ],
  "accuracy": 1
}
CaseResultInterpretation
Row split4 shared groups; 100% accuracyEvery test group has a memorized label.
Held-out-group split0 shared groups; 50% accuracyUnseen C and D receive the fallback label 0.
Actual targetNew groupsThe held-out-group result matches this example’s stated prediction question.
In [3]: Interpretation

3. Interpretation and limitations

Both splits use distinct row IDs in training and test. That fact alone misses the group relationship. The overlap matrix therefore shows group membership directly, while the accuracy chart shows what the particular memorizing rule achieves. Report both the score and the unit of separation.

A row-level split is not universally wrong. It would test a different use case, such as new records from already-known groups, if that matched the intended deployment. Here it is misleading only relative to the declared goal of unseen-group prediction. The notebook keeps that goal visible rather than calling every overlap a violation. [3]

Editorial evaluation checklist: freeze the prediction target, split before fitting transformations, use designated validation data for choices and preserve a final test for assessment. Time-dependent tasks need a suitable temporal design; a random shuffle is not automatically appropriate. [1] [2] [3]

Practice exercise

Explain why unique record IDs do not eliminate shared-group information in the row split. Then describe one deployment goal for which known-group testing could be legitimate.

In [4]: Python

4. Worked Python example

The core calculation below uses only the Python standard library. The full download adds the verification assertions and input fixtures. It runs locally on synthetic data; it does not load a model or contact a service.

lab.py · Core example · Python 3.13
LABELS = {"A": 0, "B": 1, "C": 0, "D": 1}


def evaluate(split):
    # Group IDs are features of this deliberately memorizing baseline.
    if split == "row":
        train, test = list(LABELS), list(LABELS)
    else:
        train, test = ["A", "B"], ["C", "D"]
    memory = {group: LABELS[group] for group in train}
    predictions = [memory.get(group, 0) for group in test]
    accuracy = sum(p == LABELS[g] for p, g in zip(predictions, test)) / len(test)
    return {"train": train, "test": test,
            "overlap": sorted(set(train) & set(test)),
            "predictions": predictions, "accuracy": accuracy}

Download full Python example and checks ↓

Run: python -I lab.py. Recorded run: Python 3.13; 2 fixture states plus scenario assertions passed. The JSON output is a reproducibility record, not a real-system benchmark.

In [5]: Knowledge check

5. Questions and explanations

Predict before opening the explanation. Answers refer to the stated scope and synthetic inputs.

Q1 · GroupsDoes this row split keep groups out of both train and test?
Does this row split keep groups out of both train and test?
Answer and explanation

Answer: No. Each group contributes one record to training and another to test.

Q2 · GoalWhy is the group split the relevant one for this toy’s declared target?
Why is the group split the relevant one for this toy’s declared target?
Answer and explanation

Answer: It evaluates unseen groups. The target is prediction for groups absent from training. [3]

Q3 · SelectionShould final test results guide model choices?
Should final test results guide model choices?
Answer and explanation

Answer: No; that uses test information for selection. The test should remain separate from model choices; use validation work for those decisions. [2]

Q4 · ResultWhat does the 50% result establish?
What does the 50% result establish?
Answer and explanation

Answer: The actual result of this synthetic rule and split. It is a constructed fixture with a declared fallback rule.

In [6]: Sources

References and source links

  1. Hugging Face authors. (2022). Probabilistic Time Series Forecasting with 🤗 Transformers.https://huggingface.co/blog/time-series-transformers
  2. scikit-learn maintainers. (v1.9.1). 12. Common pitfalls and recommended practices — scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/common_pitfalls.html
  3. scikit-learn maintainers. (v1.9.1). 3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/cross_validation.html
Source checks and provenance

Primary publisher pages and relevant passages checked on 13 Nov 2026. Source support is not independent expert fact verification. Main documentation can change; retrieved-body hashes, dates, version labels and saved revision IDs are in the audit download.

[1] Probabilistic Time Series Forecasting with 🤗 Transformers

Evidence: Load Dataset; Create DataLoaders. A saved forecasting example provides context for explicit data splits. Its specific temporal setup is not generalized to every task.

Public ResearchNode original c8e031ab6c189d847041872c860e5aa0
Revision 3ab8f467722743350f0c13978e6fbb96. Captured 2026-11-06.

[2] 12. Common pitfalls and recommended practices — scikit-learn 1.9.1 documentation

Evidence: 12.2 Data leakage; How to avoid data leakage. Training-only preprocessing and separation of test information; examples are not copied.

[3] 3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.9.1 documentation

Evidence: 3.1.2.4.1 Group K-fold. Group-aware evaluation for new-group generalization; not a blanket claim that all group overlap is leakage.

Claim-to-evidence ledger ↓ · Review report ↓