Researchers release Multi-Label Agreement framework for measuring annotator consistency
A new arXiv paper proposes MLA, a framework that adapts inter-annotator agreement measurement to multi-label annotation tasks, where existing kappa and alpha metrics fall short.
In plain words
When people label data, researchers need to check how consistently they agree. Standard agreement scores work for tasks with one label per item, but many tasks assign several labels at once. The paper introduces MLA, a method that compares annotators' full label sets, counting both labels they jointly chose and jointly left out.
What was announced
The authors, writing on arXiv, argue that established agreement metrics such as Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are designed for single-label settings, while multi-label tasks require extra decisions about how full label sets, joint selections, and joint omissions should be compared. MLA keeps the complete binary label vector for each item and uses average pairwise Hamming agreement, so both shared selections and shared omissions count toward the score.
The paper is candid that MLA is not an entirely new statistic: its pooled-marginal form is a chance-disagreement-weighted average of label-wise Fleiss or Scott-type coefficients, closely related to Krippendorff's alpha with normalized Hamming distance. The framework offers pooled-marginal, Gwet AC1, and fixed-baseline normalizations, reports results at item, label, and corpus level, and includes a conditional minimum-agreement diagnostic. A worked example and Monte Carlo simulations show the values depend on label prevalence, the declared label inventory, and the chosen model of expected agreement. A Python implementation is available, and the authors avoid claiming any normalization is universally best.
What it does not show
- The source is the paper's abstract only, so the worked example, simulations, and implementation details are not independently verifiable from the supplied text.
- No external evaluation or independent confirmation of the framework's claims is available in the supplied material.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.