1. Architecture and scope
An evaluation split encodes a prediction question. If the intended use is a new group, testing on another row from a training group can answer an easier question. scikit-learn documents group-aware splitting to keep a group out of both training and test simultaneously. The relevant unit might be a person, source document or other correlated group in an actual task. [3]
Data leakage uses information during model building that would not be available at prediction time. Test results should not drive model choices, and fitted preprocessing belongs on the training data. The saved forecasting tutorial also makes its train, validation and test windows explicit; its temporal design is context rather than a universal split recipe. [1] [2]
Scope. A synthetic grouped dataset and a memorizing rule. No real model is trained and no production accuracy is estimated.
Workflow and scope notes
- Define prediction. State what will be new when the system is actually used. This example targets groups absent from training.
- Choose split unit. Separate data using the unit appropriate to that prediction question. Keeping unique row IDs apart does not guarantee groups are separate.
- Fit on training. Learn or choose behavior from training data and designated validation work. The toy memorizes group labels and defaults to 0 for unseen groups.
- Evaluate once. Compute the stated metric on the held-out data and report the split design. Repeatedly choosing from test results makes those results part of model selection.
2. Row split versus held-out groups
Groups A, B, C and D each have two records. Their fixed labels are 0, 1, 0 and 1. The rule memorizes a group’s training label and predicts 0 for an unseen group. Choose the split and inspect overlap alongside accuracy.
Row split versus held-out groups
Original synthetic experiment · Change an input to update the figure.
Row split: A1 B1 C1 D1 train; A2 B2 C2 D2 test. Group split: all A/B train; all C/D test. The label rule is deliberately constructed. Its 100% and 50% scores do not estimate the effect size of leakage in any real dataset.
Calculation record
{
"inputs": {
"split": "row"
},
"train": [
"A",
"B",
"C",
"D"
],
"test": [
"A",
"B",
"C",
"D"
],
"overlap": [
"A",
"B",
"C",
"D"
],
"predictions": [
0,
1,
0,
1
],
"accuracy": 1
}| Case | Result | Interpretation |
|---|---|---|
| Row split | 4 shared groups; 100% accuracy | Every test group has a memorized label. |
| Held-out-group split | 0 shared groups; 50% accuracy | Unseen C and D receive the fallback label 0. |
| Actual target | New groups | The held-out-group result matches this example’s stated prediction question. |
3. Interpretation and limitations
Both splits use distinct row IDs in training and test. That fact alone misses the group relationship. The overlap matrix therefore shows group membership directly, while the accuracy chart shows what the particular memorizing rule achieves. Report both the score and the unit of separation.
A row-level split is not universally wrong. It would test a different use case, such as new records from already-known groups, if that matched the intended deployment. Here it is misleading only relative to the declared goal of unseen-group prediction. The notebook keeps that goal visible rather than calling every overlap a violation. [3]
Editorial evaluation checklist: freeze the prediction target, split before fitting transformations, use designated validation data for choices and preserve a final test for assessment. Time-dependent tasks need a suitable temporal design; a random shuffle is not automatically appropriate. [1] [2] [3]
Explain why unique record IDs do not eliminate shared-group information in the row split. Then describe one deployment goal for which known-group testing could be legitimate.
4. Worked Python example
The core calculation below uses only the Python standard library. The full download adds the verification assertions and input fixtures. It runs locally on synthetic data; it does not load a model or contact a service.
LABELS = {"A": 0, "B": 1, "C": 0, "D": 1}
def evaluate(split):
# Group IDs are features of this deliberately memorizing baseline.
if split == "row":
train, test = list(LABELS), list(LABELS)
else:
train, test = ["A", "B"], ["C", "D"]
memory = {group: LABELS[group] for group in train}
predictions = [memory.get(group, 0) for group in test]
accuracy = sum(p == LABELS[g] for p, g in zip(predictions, test)) / len(test)
return {"train": train, "test": test,
"overlap": sorted(set(train) & set(test)),
"predictions": predictions, "accuracy": accuracy}
Download full Python example and checks ↓
Run: python -I lab.py. Recorded run: Python 3.13; 2 fixture states plus scenario assertions passed. The JSON output is a reproducibility record, not a real-system benchmark.
5. Questions and explanations
Predict before opening the explanation. Answers refer to the stated scope and synthetic inputs.
References and source links
- Hugging Face authors. (2022). Probabilistic Time Series Forecasting with 🤗 Transformers.https://huggingface.co/blog/time-series-transformers
- scikit-learn maintainers. (v1.9.1). 12. Common pitfalls and recommended practices — scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/common_pitfalls.html
- scikit-learn maintainers. (v1.9.1). 3.1. Cross-validation: evaluating estimator performance — scikit-learn 1.9.1 documentation.https://scikit-learn.org/stable/modules/cross_validation.html
Source checks and provenance
Primary publisher pages and relevant passages checked on 13 Nov 2026. Source support is not independent expert fact verification. Main documentation can change; retrieved-body hashes, dates, version labels and saved revision IDs are in the audit download.
Evidence: Load Dataset; Create DataLoaders. A saved forecasting example provides context for explicit data splits. Its specific temporal setup is not generalized to every task.
Public ResearchNode original c8e031ab6c189d847041872c860e5aa0
Revision 3ab8f467722743350f0c13978e6fbb96. Captured 2026-11-06.
Evidence: 12.2 Data leakage; How to avoid data leakage. Training-only preprocessing and separation of test information; examples are not copied.
Evidence: 3.1.2.4.1 Group K-fold. Group-aware evaluation for new-group generalization; not a blanket claim that all group overlap is leakage.