Researchers extend algebraic evaluation framework to data streams with delayed labels
An arXiv paper describes a way to keep scoring model performance on streams when labels arrive late, partially, or not at all, with Lean 4-verified statements and an open package.
In plain words
When a model makes predictions on a continuous flow of data, the true answers often show up late or only for some cases, which can distort performance scores. The paper adapts an existing algebraic evaluation method so scores update over time while treating missing labels as an observation problem. The authors also verify key mathematical claims in the Lean 4 proof assistant.
What was announced
The framework, presented in an arXiv abstract by its authors, extends an algebraic approach that derives pooled (micro) and per-class (macro) measures from a single confusion matrix to streaming settings. Counts are weighted in time using exponential fading or a sliding window, with an update cost of O(m^2) per step regardless of stream length, and each pooled measure remains a weighted combination of the per-class ones. Delayed or selective feedback is handled through an observation mask: until a label arrives, an instance is counted using the model's calibrated class probabilities, and the bias of this interim estimate depends on their calibration error.
The authors report that the method separates prior shift from concept drift by monitoring prior-free coordinates such as recall and specificity, attributing changes to the two sources via a first-order Jacobian decomposition. On synthetic streams and four real datasets, they state that a detector on these coordinates with per-class memory raised no excess false alarms under label shift, unlike Jacobian-alarm and error-rate detectors. They also note the delayed-label correction recovered recall in some real settings but was only partial for precision, specificity and F1. Twenty-five statements are checked in Lean 4, and the code is available in the aggmetrics package.
What it does not show
- The evidence is limited to the paper's abstract; no independent evaluation, full methodology, or peer review is available from the supplied passages.
- The reported benefits, such as recall recovery and false-alarm behavior, are the authors' own claims on their chosen synthetic and real datasets and are not independently confirmed.
For security and IT teams
- Status
- The code is available in the aggmetrics package, and twenty-five statements are checked in Lean 4, according to the abstract.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.