{"sources":[{"id":"S1","url":"https://scikit-learn.org/stable/modules/calibration.html","title":"Probability calibration","publisher":"scikit-learn developers","published_at":null,"accessed_at":"2026-11-14","version":"1.9.1 as displayed at access","document_id":"","revision_id":"","limitations":"Mutable, undated stable documentation. Supports definitions, grouped coordinates, scoring caveat and independent calibration data; not the invented sample's results."},{"id":"S2","url":"https://proceedings.mlr.press/v70/guo17a/guo17a.pdf","title":"On Calibration of Modern Neural Networks","publisher":"Proceedings of Machine Learning Research","published_at":"2017","accessed_at":"2026-11-14","version":"PMLR 70:1321–1330, conference paper; year confirmed on publisher landing page","document_id":"","revision_id":"","limitations":"Used for selected-label confidence and empirical approximation. No historical benchmark conclusions are generalized to current models; exact publication day is not asserted."},{"id":"S3","url":"https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html","title":"brier_score_loss","publisher":"scikit-learn developers","published_at":null,"accessed_at":"2026-11-14","version":"1.9.1 as displayed at access","document_id":"","revision_id":"","limitations":"Mutable, undated stable documentation. Used for the binary formula and normalization; the lab does not call scikit-learn."}],"claims":[{"id":"C1","text":"Binary calibration compares positive-class probabilities with positive frequencies; reliability coordinates use group mean probabilities and positive fractions.","location":"probability-meaning; worked-sample","evidence":[{"source_id":"S1","passage":"fraction of positives","location":"Section 1.16.1, calibration curves, grouped-coordinate explanation"}],"limitations":"Supported within binary scope. Group frequencies are empirical summaries, not individual or population guarantees."},{"id":"C2","text":"Selected-label confidence concerns correctness, whereas positive-class probability always refers to class 1; finite samples motivate empirical approximations.","location":"probability-meaning; questions Q1","evidence":[{"source_id":"S2","passage":"probability of correctness","location":"Section 2, page 2, definition of associated confidence"},{"source_id":"S2","passage":"empirical approximations","location":"Section 2, page 2, after equation 1"}],"limitations":"The complement 1−0.1=0.9 is original binary arithmetic under the specified threshold. No generated-answer calibration claim."},{"id":"C3","text":"Four ten-document groups with probabilities 0.1, 0.3, 0.7, 0.9 and positive counts 2, 4, 6, 8 are synthetic teaching assumptions.","location":"probability-meaning; worked-sample descriptions; python-example; lab.py","evidence":[],"limitations":"Author-defined inputs, not empirical claims. Every individual label is zero or one. No external source is offered as evidence for invented documents.","kind":"original_example","basis":"Author-defined inputs, not empirical claims. Every individual label is zero or one. No external source is offered as evidence for invented documents."},{"id":"C4","text":"The example has accuracy 0.70, prevalence and mean probability 0.50, four absolute group gaps of 0.10, and zero pooled gap.","location":"worked-sample; questions Q2; python-example; lab.py","evidence":[{"source_id":"S1","passage":"average predicted probability in each bin","location":"Section 1.16.1"}],"limitations":"Original arithmetic on C3: 8+6+6+8=28 correct; 2+4+6+8=20 positives; equal-size mean p=0.5. The explicitly defined grouped gap is not a claim about every calibration-error implementation."},{"id":"C5","text":"Binary Brier loss is mean squared positive-class probability error, ranges from zero to one, and equals half the unscaled two-class sum.","location":"brier-calculation; python-example; lab.py","evidence":[{"source_id":"S3","passage":"the Brier score is usually divided by two","location":"Notes, binary classification paragraph and displayed binary formula"}],"limitations":"Supported binary normalization. Endpoints follow from binary labels and probabilities in [0,1]."},{"id":"C6","text":"Original Brier scores average to 0.21; replacing probabilities by observed fractions gives 0.20 without changing accuracy. Group loss equals r(1−r)+(p−r)^2. The hypothetical fresh D group reverses which probability has lower loss.","location":"brier-calculation; evaluation-design; questions Q3; python-example; lab.py","evidence":[{"source_id":"S3","passage":"the predicted probability of the positive class","location":"Notes, explanation of binary formula"}],"limitations":"Original derivation from C3 and C5. Before-mapping group sums: 1.7,2.5,2.5,1.7; after: 1.6,2.4,2.4,1.6. Divide their totals by 40. Expanding r(1−p)^2+(1−r)p^2 yields r−2rp+p^2. Fresh D with r=0.9 gives 0.09 at p=0.9 and 0.10 at p=0.8. No execution or generalization result."},{"id":"C7","text":"Lower Brier loss alone does not establish better calibration because the score also reflects discrimination and outcome uncertainty.","location":"brier-calculation; questions Q4","evidence":[{"source_id":"S1","passage":"does not necessarily mean a better calibrated model","location":"Section 1.16, scoring-rule note"}],"limitations":"Supported caveat; no decomposition estimator is implemented."},{"id":"C8","text":"Calibrator fitting should use data independent of classifier fitting; a further untouched evaluation sample is proposed here.","location":"evaluation-design; questions Q5","evidence":[{"source_id":"S1","passage":"independent of the training data","location":"Section 1.16.2"}],"limitations":"Source supports classifier/calibrator data separation. The additional evaluation sample and reporting protocol are author-proposed, not an executed study or an attributed mandatory three-way split."},{"id":"C9","text":"The supplied code groups exact repeated probabilities and constructs replacements from the same labels it evaluates; these calculations cannot validate generalization.","location":"python-example; evaluation-design; questions Q5; lab.py","evidence":[],"limitations":"Direct static inspection of original package code: make_rows(mapped=True) uses positives/count, and summarize regroups by p. Intended assertions are not execution evidence. The proposed fresh-outcome and input-change exercises are unperformed. Standard-library import and absence of file/network/model calls are directly inspectable.","kind":"original_example","basis":"Inspect make_rows(mapped=True) and summarize in lab.py, plus the isolated execution report; the code reuses the constructed labels, so it cannot establish generalization."},{"id":"C10","text":"The lesson cites three live public primary sources and uses synthetic examples; it makes no claim of independent expert fact verification.","location":"references; source-checks","evidence":[],"limitations":"Automated source-support review is not independent expert verification. Mutable documentation may change after the recorded access date.","kind":"process","basis":"Inspect the three bibliography entries, the synthetic inputs in lab.py and the source-check disclosure. No saved original is cited as primary support."}]}