← News

ResearchSources: arXiv · 18 Nov 2026

arXiv study: macro-F1 can overstate tail recall in multi-label classifiers

A new diagnostic estimates, before any model is trained, when rare-label recall depends on frequent co-occurring labels, and finds the gap on six of 23 corpora.

In plain words

In multi-label classification, a rare category is often predicted correctly only because it usually appears alongside common ones. The authors introduce a way to measure, from training labels alone, how much reported recall could depend on those companions. Their checks suggest headline scores can credit tail hits that would disappear without companions.

What was announced

The paper's authors argue that existing diagnostics of this effect require a trained model, so none can say before training whether a dataset admits the failure. Because macro-F1 is built from per-label confusion counts, it does not change with how a label's recall splits between occurrences with and without a companion label. The team proposes a within-label estimand of that division, a reading rule under which dependence counts as falling only if companion-free recall rises, and a statistic computed from training labels that bounds how far reported recall can exceed companion-free recall before any training happens.

Across 23 corpora, seven training objectives and five families of label-correlation mechanisms, the authors report that companion-free recall fell short on all six corpora where the estimand was defined. They state that up to 18.4% of the tail true positives credited by macro-F1 would vanish at the companion-free rate, staying within the training-label bound on every corpus. No long-tail objective in their tests reduced the dependence, though architecture did: models whose tail score is computed from other labels' predictions were strongly dependent. The work is an arXiv abstract, so the figures await review and independent confirmation.

What it does not show

  • Evidence is limited to the paper's abstract; full methods, datasets and results are not available for verification.
  • The six of 23 corpora result applies only where the estimand is defined, so coverage of the other corpora is unclear from the abstract.
  • The findings are the authors' own measurements and have not been independently confirmed or peer reviewed.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.