Probability estimates and observed frequencies
A binary classifier assigns a probability p to a positive outcome, represented by y = 1. Calibration concerns agreement between assigned probabilities and observed positive frequencies across groups. A group assigned probabilities around 0.8 should have a positive fraction around 0.8. This is a group interpretation, not a promise about one outcome. [1]
Positive-class probability differs from confidence in the selected label. Suppose our decision rule selects class 1 when p ≥ 0.5. A prediction of p = 0.1 then selects class 0, whose assigned probability is 0.9. Guo and colleagues define classification confidence through correctness of the selected class. Keep that definition separate from the positive-label frequencies used here. [2]
Our fictional application predicts whether a document belongs in a topic collection. All documents, labels and probabilities below are invented teaching inputs. The objective is to calculate what a probability report establishes and identify the evidence it lacks. No classifier is trained and no real model performance is reported.
Scope and limits: this notebook concerns binary positive-class probabilities on one constructed sample. It does not establish population calibration, reliability within demographic subgroups, robustness after distribution change, or truthfulness of generated answers. Finite-sample summaries are empirical approximations, not direct knowledge of the population relationship between predictions and outcomes. [2]
A forty-document worked example
Assume four groups, each containing ten documents with one repeated assigned probability. The positive counts are chosen for transparent arithmetic rather than sampled from a population. Each group description represents ten separate binary observations; a positive fraction is a summary, not a fractional outcome label.
- Group A
- Assigned probability 0.10; 2 positives out of 10; observed positive fraction 0.20.
- Group B
- Assigned probability 0.30; 4 positives out of 10; observed positive fraction 0.40.
- Group C
- Assigned probability 0.70; 6 positives out of 10; observed positive fraction 0.60.
- Group D
- Assigned probability 0.90; 8 positives out of 10; observed positive fraction 0.80.
At the 0.5 threshold, A and B receive negative predictions while C and D receive positive predictions. There are 8 + 6 + 6 + 8 = 28 correct decisions among 40 documents, so accuracy is 0.70. The mean assigned probability is 0.50, matching the overall positive fraction of 20/40. Nevertheless, every group has a discrepancy of ten percentage points.
The low-probability groups underestimate their observed positive fractions, while the high-probability groups overestimate theirs. Equal group sizes allow these signed errors to cancel. Define a descriptive grouped gap as the count-weighted average of |assigned probability − positive fraction|. For these four groups it is 0.10. Pooling everything into one group instead gives a gap of zero. Neither number changes any document's prediction; the grouping changes the question being summarized.
A reliability plot uses grouped mean probabilities and positive fractions as coordinates. Our four descriptions supply those coordinates and their counts directly. [1] Here, prose is sufficient to expose the cancellation without a separate diagram. In a report based on this example, state the grouping rule alongside the gap so that readers can reproduce it.
Binary Brier score and sample arithmetic
We use the binary Brier convention: Brier = Σ(pᵢ − yᵢ)²/N. It averages squared errors between positive-class probabilities and binary labels. Values range from zero to one, with smaller values better. Summing squared errors for both class probabilities without halving would give twice this value, so the normalization must be stated. [3]
In group A, two positive labels contribute 2 × (0.1 − 1)² = 1.62. Eight negative labels contribute 8 × 0.1² = 0.08. Their sum divided by ten is 0.17. Repeating this calculation gives group scores 0.17, 0.25, 0.25 and 0.17. Equal group sizes make the overall score their average: 0.21. These values are original arithmetic, not captured program output.
Now replace the assigned probabilities 0.1, 0.3, 0.7 and 0.9 with the corresponding observed fractions 0.2, 0.4, 0.6 and 0.8. No probability crosses the decision threshold, so accuracy remains 0.70. The group Brier scores become 0.16, 0.24, 0.24 and 0.16, giving an overall score of 0.20. A probability adjustment can therefore change this score without changing a single classification decision.
The algebra explains the exact improvement. If a group has positive fraction r and every member receives probability p, its mean squared error is r(1 − p)² + (1 − r)p² = r(1 − r) + (p − r)². Holding the observed labels fixed leaves the first term unchanged. Each original probability differs from its group's fraction by 0.1, adding 0.01. Substituting p = r removes that term. This establishes a sample identity, not a claim that the observed fraction is the probability for future documents.
Brier loss also reflects discrimination and outcome uncertainty. A lower score alone therefore does not establish better calibration. Read it alongside grouped frequencies and counts. [1]
Illustrative Python calculation
The original lab below expands the four groups into individual observations. It is synthetic and illustrative: it has not been executed during authoring. Assertions encode expected arithmetic rather than a passed test result. It uses only the Python standard library and contains no file, network or model calls. Download the full lab.py.
from math import isclose
# Illustrative synthetic inputs; not execution output.
GROUPS = (
(0.1, 10, 2),
(0.3, 10, 4),
(0.7, 10, 6),
(0.9, 10, 8),
)
def make_rows(mapped=False):
rows = []
for probability, count, positives in GROUPS:
p = probability
if mapped:
p = positives / count
for i in range(count):
rows.append((p, int(i < positives)))
return rows
def summarize(rows):
n = len(rows)
correct = sum(
int(p >= 0.5) == y for p, y in rows
)
brier = sum((p - y) ** 2 for p, y in rows) / n
groups = {}
for p, y in rows:
groups.setdefault(p, []).append(y)
gap = 0.0
for p, labels in groups.items():
fraction = sum(labels) / len(labels)
gap += len(labels) * abs(p - fraction)
return correct / n, brier, gap / n
def main():
cases = (
(False, (0.70, 0.21, 0.10)),
(True, (0.70, 0.20, 0.00)),
)
for mapped, expected in cases:
actual = summarize(make_rows(mapped))
for value, target in zip(actual, expected):
assert isclose(value, target, abs_tol=1e-12)
print("mapped:", mapped)
for name, value in zip(
("accuracy", "brier", "grouped_gap"), actual
):
print(name, round(value, 2))
if __name__ == "__main__":
main()
The grouping key is an exact repeated probability. This is appropriate for the four deliberately repeated values, but is not a general binning method for continuous predictions. The mapping also reads the same labels later used for evaluation. Its zero grouped gap is built into its construction; it cannot demonstrate successful generalization.
Separating construction from evaluation
Calibration fitting should use data independent of classifier fitting. The scikit-learn guide explains that training predictions can bias a fitted calibrator. [1] For an extension of this exercise, reserve an additional untouched evaluation sample after selecting the probability mapping. This third-set arrangement is a proposed experimental design, not an experiment performed here.
Keep the four original probability values and freeze their replacements before inspecting new labels. Collect fresh labeled documents for each group, then report both versions' accuracy, Brier score, group counts and observed fractions. If a group has no new documents, record the missing evidence rather than assigning it a positive fraction. Decide these reporting choices before looking at which result appears most favorable.
Group D illustrates the missing evidence. Its original ten documents contain eight positives, motivating the replacement of 0.9 by 0.8. If a fresh group of ten instead contained nine positives, the original probability would match that new fraction while the replacement would not. In that hypothetical group, the original Brier score would be 0.09 and the mapped score 0.10. Both the original sample and this extension are synthetic; neither predicts which outcomes will actually arrive.
As a further exercise, change one positive count in the lab and recalculate the expected assertions before running it. Explain separately what changed in the threshold decisions, squared probability errors and grouped gap. Reusing old expected values after changing the inputs would test the wrong arithmetic. This exercise investigates the calculation, not a real calibrator's usefulness.
Explanatory questions
Q1 · ProbabilityWhat does p = 0.1 mean for a negative decision?
The positive class receives probability 0.1 and the negative class receives 0.9. Our rule selects the negative label. Its confidence is therefore 0.9, while the positive-class probability remains 0.1. [2]
Q2 · AggregationWhy does the matching overall mean conceal errors?
The two low-probability groups underestimate their fractions by 0.1; the two high-probability groups overestimate by 0.1. Equal group sizes cancel the signed discrepancies. Separate groups expose them.
Q3 · AccuracyWhy does accuracy stay fixed after the mapping?
No probability crosses 0.5, so every selected label stays unchanged. Squared probability errors still change: the sample Brier score falls from 0.21 to 0.20 while accuracy remains 0.70.
Q4 · ScoringDoes lower Brier loss establish better calibration?
No. The score also reflects discrimination and outcome uncertainty. Include grouped frequencies and counts when interpreting it. [1]
Q5 · EvaluationWhy does the mapped sample fail to validate the mapping?
Its labels both define and assess the replacement probabilities. Matching the observed fractions is guaranteed by construction. Evaluate frozen replacements on fresh outcomes to investigate performance beyond that construction.
Bibliography
References and source links
- scikit-learn developers · documentation 1.9.1, undated. Probability calibration.
https://scikit-learn.org/stable/modules/calibration.html - Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. · 2017. On Calibration of Modern Neural Networks. PMLR 70, 1321–1330.
https://proceedings.mlr.press/v70/guo17a/guo17a.pdf - scikit-learn developers · documentation 1.9.1, undated. brier_score_loss.
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html
Source-check disclosure
Evidence scope and remaining checks
No saved originals are cited as evidence for this lesson. This statement concerns the selected citations, not the existence of library search results. The three public primary sources above were opened on 14 November 2026 and their relevant passages read. The documentation pages displayed version 1.9.1; their stable URLs are mutable. The paper supplies definitions, not evidence about current models.
The ledger separates sourced premises from synthetic inputs, original arithmetic and implementation inspection. Automated source-support checks do not constitute independent expert fact verification. Examples use synthetic data and do not measure any deployed model.