← News

ResearchSources: arXiv · 19 Nov 2026

Audit of deployed CV priority sorter finds evaluation flaws and a hidden cost in active learning

Researchers audited their own academic hiring tool, which orders CVs for committee review, and report three findings that changed the product, including a flaw in a common agreement score.

In plain words

A team built a tool that suggests the order in which a hiring committee should read CVs, while leaving every decision to the committee. They then audited their own evaluation of the tool and found problems: the test data was too easy, a reported score was misleading, and a proposed feature would have increased the committee's reading workload.

What was announced

The system, deployed on a national academic vacancy platform, pairs a frozen multilingual transformer encoder with a small per-session linear model that retrains after each committee label and is discarded when the session ends. The authors report that on a pool of 300 synthetic CVs, a committee following the tool's order reached 95% of relevant candidates after reading half the pile, while an arbitrary order needed nearly the whole pile. The audit itself, not the system, is the paper's focus.

Three audit findings changed the product. First, the test pool could be sorted by document length alone, making precision-style metrics uninformative; regenerating it under a shared length budget removed that shortcut while the reading-depth result held. Second, the four-level agreement score such systems report is a binary probability cut at fixed thresholds, so it tracks classifier confidence rather than ranking quality and falls as labels accumulate. Third, uncertainty sampling for a "suggest the next CV" button preserved model quality but raised total committee reading above the no-label baseline, because the CVs it selects are those that would otherwise be skipped. Separately, a hosted LLM embedding model lifted zero-label ordinal discrimination from 0.66 to 0.80 under length control, a gain a local cross-encoder matched without sending data out.

What it does not show

  • The evidence is the paper's abstract only, so the full methodology, dataset details and reported fixes cannot be verified from this excerpt.
  • The 95% reading-depth result is measured on synthetic CVs, and the abstract does not report performance on real applications.
  • The abstract does not name the platform, describe deployment scale, or state whether the audit findings have been independently confirmed.

For security and IT teams

Access
The paper reports that a local cross-encoder matched the cold-start gain of a hosted LLM embedding model without sending data out, which may matter where CV data cannot leave the premises.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.