← News

ResearchSources: arXiv · 18 Nov 2026

Framework proposes standardized measurement of expert-AI interactions for prospective risk detection

An arXiv concept paper outlines a way to compress expert-AI interactions into comparable fields, enabling population-level risk monitoring without access to model internals.

In plain words

A researcher has proposed a framework that turns exchanges between experts and AI systems into structured, standardized records that can be compared and scored. The idea, called "AI epidemiology", is to detect risks across deployed systems by looking for patterns in these scores, similar to how epidemiology studies disease patterns in populations. This is a proposal with a testing protocol, not a finished, validated system.

What was announced

The paper, published on arXiv, defines a grammar of eight interaction fields and a statistical protocol that includes paired bootstrap inference, DeLong's test as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm-Bonferroni correction. The author argues that large language models, under bounded conditions, could reliably produce standardized assessments of the evidential and policy alignment of expert-AI interactions, giving experts an immediate signal during deployment and institutions a basis for monitoring alignment patterns across mission types, models, and domains.

In a minimal application of the protocol to a published expert-AI corpus, the judge reproduced its policy and evidential alignment scores across two runs under the specified conditions. The author stresses that judge reliability at scale remains to be validated, and that population-level claims belong to a staged research program rather than to results claimed in the paper itself.

What it does not show

  • The evidence is limited to the paper's abstract; the full article was not reviewed, so details of the framework's implementation and testing are not visible here.
  • The only empirical result reported is score reproduction across two runs on a single published corpus; reliability at scale and any association with downstream outcomes remain unvalidated.
  • The paper is a concept paper; no independent confirmation of the framework or its claims is available.

For security and IT teams

Scope
The framework is designed to work without access to model internals, meaning it operates on interaction records rather than internal model data.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.