← Learn / Notebooks

AI securityPractitionerStudy guide

Membership Inference: False Positives and Base Rates

Evaluate training-data membership claims using confusion counts, low false-positive rates and explicit assumptions about candidate populations.

Before you start: You should understand training and test datasets, binary classification and basic probability.

AI-assisted study notebook · Published 2026-11-15. Separate automated source and visual reviews passed; these checks are not independent expert fact verification. Source audit · Review record.

Training-data membership as a privacy question

Membership inference asks whether a particular record was used to train a model. The candidate record is supplied to the attacker; the unknown is its membership. Shokri and colleagues demonstrated attacks using prediction outputs from black-box classifiers. Their experiments establish a possible information channel, not a success rate that transfers to every model. [1]

Their shadow-model approach trains auxiliary models with known training membership, then uses those models' outputs to train a membership classifier. This separates the target model's ordinary prediction task from the attacker's membership prediction task. [1]

Carlini and colleagues recommend evaluating membership attacks at low false-positive rates: aggregate accuracy can obscure whether an attack confidently identifies members. This notebook explores the evaluation problem through original arithmetic. [2]

Scope and limits. The worked example contains invented counts and no personal records, model queries or trained attack. It teaches interpretation of a hypothetical detector at a fixed threshold. It does not establish that a named model leaks data, determine anyone's training membership or certify privacy. A table communicates the count relationships directly; no diagram is needed.

Define the experiment before interpreting a score

For this exercise, define one candidate as one distinct record and membership relative to one frozen training dataset. Assume an already selected detector emits positive, meaning predicted member, or negative. Actual membership is available to the evaluator. These are teaching assumptions; the lab does not implement the detector or assume that its scores are probabilities.

An evaluation record for this hypothetical detector should specify the model version, membership unit, candidate selection rule, information available to the detector and threshold. A record-level result cannot directly answer a person-level question when one person contributes several records: the two questions count different units. Write the unit beside each denominator.

In this proposed exercise, select the threshold using a development collection, freeze it, then count predictions on a distinct assessment collection. Otherwise the counts describe a threshold selected with knowledge of those same answers. This is a design recommendation, not a claim that the synthetic counts were obtained through that procedure.

The four outcomes are true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). Members total TP + FN; nonmembers total FP + TN. True-positive rate is TP/(TP + FN), while false-positive rate is FP/(FP + TN). Precision divides TP by all positive predictions, TP + FP. These definitions require nonzero denominators.

A false-positive rate of 1% means one positive prediction per hundred actual nonmembers on average under the assumed rates. It does not mean that 1% of positive predictions are wrong. That fraction is FP/(TP + FP), which also depends on the proportion of members among the candidates. This denominator change drives the worked example.

Worked example with two candidate populations

Assume TPR = 0.60 and FPR = 0.01 at a fixed threshold. Construct a balanced collection of 1,000 members and 1,000 nonmembers. Assign exactly 600 true positives and 10 false positives. The remaining counts are 400 false negatives and 990 true negatives. These deliberately chosen integers imply 98.36% precision and 79.50% accuracy.

Next construct a collection with the same 1,000 members but 99,000 nonmembers. Hold both conditional rates constant as an explicit assumption. True positives remain 600; false positives become 990. Only 600 of the 1,590 positive predictions are members, giving approximately 37.74% precision. The interpretation changes despite identical TPR and FPR.

Synthetic fixed-rate populations; counts are constructed, not measured
QuantityBalanced collection1% membership collection
Members / nonmembers1,000 / 1,0001,000 / 99,000
TP / FN600 / 400600 / 400
FP / TN10 / 990990 / 98,010
Precision98.36%37.74%
Accuracy79.50%98.61%

Accuracy rises because correct nonmember predictions dominate the second collection. An always-negative detector would achieve 99% accuracy there while identifying no members. The apparent improvement from 79.50% to 98.61% therefore cannot be read as better member identification. The conditional rates were held fixed.

Let p be membership prevalence, t the TPR and f the FPR. For N candidates, expected true positives equal Npt and expected false positives equal N(1 − p)f. Dividing the first by their sum cancels N: precision = pt/[pt + (1 − p)f], provided the denominator is positive. This is a population-rate relationship; finite samples need not match the expected counts exactly.

At p = 0.01 and t = 0.60, at least 90% precision requires f ≤ pt(1 − 0.90)/[0.90(1 − p)] = 1/1485, approximately 0.0006734, or 0.06734%. This conditional requirement does not identify a usable threshold. Changing a real detector's threshold can change t along with f; a detector need not attain the assumed pair.

The comparison assumes conditional rates transfer between candidate collections. If candidate selection changes the kinds of records assessed, that assumption needs evidence. Unknown membership prevalence should be explored as an uncertainty, not silently replaced with 50%. In this exercise, changing p while holding t and f fixed isolates the effect of prevalence; it does not simulate every consequence of changing the candidate population.

Illustrative Python calculation

This original Python 3 example is illustrative and synthetic. It uses exact rational arithmetic, then prints rounded percentages. It contains no shell, file, network or model calls. Its assertions check arithmetic relationships; even successful execution cannot validate a membership detector because none is implemented. Download lab.py.

from fractions import Fraction


def precision(prevalence, tpr, fpr):
    positive_mass = prevalence * tpr + (1 - prevalence) * fpr
    if positive_mass == 0:
        return None
    return prevalence * tpr / positive_mass


def counts(members, nonmembers, tpr, fpr):
    tp = members * tpr
    fp = nonmembers * fpr
    return tp, fp, members - tp, nonmembers - fp


def main():
    # Illustrative synthetic rates, not model measurements.
    tpr = Fraction(3, 5)
    fpr = Fraction(1, 100)
    for members, nonmembers in ((1000, 1000), (1000, 99000)):
        tp, fp, fn, tn = counts(members, nonmembers, tpr, fpr)
        p = Fraction(members, members + nonmembers)
        ppv = precision(p, tpr, fpr)
        accuracy = (tp + tn) / (members + nonmembers)
        assert ppv == tp / (tp + fp)
        print(f"members={members}, nonmembers={nonmembers}")
        print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
        print(f"precision={float(ppv):.2%}")
        print(f"accuracy={float(accuracy):.2%}")

    assert precision(Fraction(1, 100), tpr, fpr) == Fraction(20, 53)
    assert precision(Fraction(1, 2), 0, 0) is None
    p = Fraction(1, 100)
    target = Fraction(9, 10)
    max_fpr = p * tpr * (1 - target) / (target * (1 - p))
    assert max_fpr == Fraction(1, 1485)
    print(f"conditional FPR ceiling={float(max_fpr):.5%}")


if __name__ == "__main__":
    main()

The helpers assume nonnegative counts and rates between zero and one; they are not an input-validation library. For arbitrary rates, returned counts can be fractional expected counts. The chosen inputs give integers. Change only the candidate counts and predict which conditional rates remain fixed. Then change fpr and observe that this artificial calculation holds tpr constant by construction.

Report what the evaluation can establish

Report TPR at stated low-FPR operating points alongside the ROC curve, which shows the threshold tradeoff. Carlini and colleagues recommend logarithmic ROC plots to expose the low-FPR region. An aggregate accuracy score alone does not describe that region. [2]

Include the counts. With 100 assessed nonmembers, one false positive changes empirical FPR by 1 percentage point. With 10,000, it changes FPR by 0.01 percentage points. Extra decimal places cannot create finer count resolution. Neither calculation alone determines an uncertainty interval.

Zero observed false positives does not imply zero population FPR. Under independent trials sharing a fixed true FPR f, the probability of observing none in n nonmembers is (1 − f)n. At f = 0.001 and n = 100, it is approximately 90.48%. A small assessment can therefore observe no errors despite a positive underlying rate. Independence and a common error probability are assumptions here, and this calculation is not a confidence interval.

For the hypothetical assessment, report TP, FP, FN and TN at the frozen threshold on the specified candidate collection. State separately any prevalence used to project precision. Observations belong to the assessed collection; projections require additional assumptions. Failure of one detector alone does not establish that other detectors fail or provide a universal privacy guarantee.

A practical exercise report can therefore have two short parts: the fixed experimental definition and observed counts, followed by conditional projections over plausible prevalence values. Keep the unit, threshold and candidate selection visible in both. If those assumptions change between comparisons, explain the change before attributing differences to detector quality.

Explanatory questions

Q1 · TargetWhat is unknown in the membership experiment?

Whether the supplied candidate record belongs to the frozen training dataset. The hypothetical evaluator knows the answer to score predictions. The exercise does not recover unknown training records. [1]

Q2 · DenominatorsWhy can 1% FPR coexist with 37.74% precision?

FPR divides 990 false positives by 99,000 nonmembers. Precision divides 600 true positives by 1,590 positive predictions. Different denominators answer different questions.

Q3 · AccuracyWhy is 98.61% accuracy insufficient here?

Nonmembers dominate the collection. An always-negative detector achieves 99% accuracy while identifying no members. The hypothetical detector identifies 600 members, so accuracy alone does not describe its behavior.

Q4 · AssumptionsDoes the calculated FPR ceiling identify a usable threshold?

No. It is conditional on 1% prevalence and 60% TPR. A threshold change may reduce TPR as well as FPR, and prevalence may be unknown.

Q5 · EvidenceWhat does zero false positives in 100 trials establish?

An empirical count of zero for those trials. Under the independent-trial assumptions, a positive FPR of 0.1% still gives about a 90.48% chance of that observation. It does not establish zero population risk.

Sources and evidence scope

References and source links

  1. Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017; v2). Membership Inference Attacks against Machine Learning Models. IEEE Symposium on Security and Privacy / arXiv.
    https://arxiv.org/html/1610.05820v2
  2. Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramèr, F. (2022; v2). Membership Inference Attacks From First Principles. arXiv.
    https://arxiv.org/html/2112.03570v2
Source-check disclosure

The evidence scope is the first paper's abstract and introduction for the membership definition and shadow-model mechanism, and Section III-B of the second for evaluation methodology. Version dates refer to the arXiv submission histories. These sources support the concepts, not the invented rates.

The accompanying ledger distinguishes source-backed statements, original arithmetic, hypothetical design choices and process records. Source reading does not constitute independent expert verification. Arithmetic checks assess the calculation; they do not assess a trained detector or establish a model's privacy. Neither cited paper measures the hypothetical detector used here.