Researchers propose protocol-based evaluation for surgical planning
An arXiv paper argues that judging surgical plans by similarity to one recorded operation is flawed, and proposes expert-defined protocol rules as the standard instead.
In plain words
Surgical plans made by AI are usually scored by how closely they match one recorded operation. The authors say this mixes up finishing a task with following the clinical protocol, since the recorded operation itself may break the rules. They propose scoring plans against expert-defined protocol rules instead.
What was announced
The paper, posted on arXiv, states that current evaluation of surgical planning borrows from activity anticipation benchmarks such as Ego4D and scores plans by trajectory similarity to a single observed execution. The authors argue this conflates task completion with protocol adherence, because the reference trajectory is only one execution that may itself deviate from the protocol. Their framework instead measures plans directly against expert-defined protocol rules.
Under that framework, the authors report that similarity-based metrics systematically fail: they penalize protocol-adherent plans that differ from the reference while rewarding protocol-violating plans that look superficially similar. When evaluating Video-LLMs, they identify two separable failure modes, unreliable video-to-state grounding and under-constrained planning. Neither is resolved by model scaling alone, while structural knowledge injection gives consistent gains. The work is an abstract, so full methods and results are not available for verification.
What it does not show
- The source is only the paper's abstract, so the framework's details, experiments and results cannot be independently checked from this excerpt.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.