RehearsalBench tests whether AI models can forecast the impact of organizational changes
A new benchmark from arXiv places AI models in a simulated bank contact center and scores their plans and predictions in dollars, finding capable forecasts alongside shared blind spots.
In plain words
Researchers built a test called RehearsalBench that asks AI models to act like a manager: diagnose how an organization runs, propose a change, and predict what will happen. The models' choices and forecasts are then checked against many simulated outcomes. The study reports that models often get the direction of effects right but misjudge how large those effects are.
What was announced
The benchmark, described in a paper on arXiv, sets each model inside a simulated bank contact center built from real call-level records. Models see only what a manager would see, then submit a plan and forecasts that are played out across 200 seeded futures and three versions of the organization. Decisions are scored in dollars at stakeholder-stated valuations, and forecasts are judged against the effect of doing nothing. Ten models made 280 attempts across 14 organizations. The strongest performer, GPT-6 Astra, beat the no-change option by 6.8% of the no-change cost and cut forecast error by 77% versus a forecaster that always predicts no effect.
The authors report that every model, on average, correctly forecast the direction of each specified effect, yet models made similar sizing mistakes, overstating a voice agent's effect on repeat contacts roughly five-fold. None of the models ever advised waiting when doing nothing was the right call. In an external check, twelve models forecast 33 real organizational changes with known outcomes, and the benchmark's ranking agreed with the real-world ranking under pre-set verdict rules, with a Spearman correlation of 0.92 and a 95% lower bound of 0.50. The authors pair this with an inconclusive first test, analysis choices made after outcomes were seen, and evidence that part of the agreement reflects how well models size their uncertainty.
What it does not show
- The source is the paper's abstract only, so the full methodology and results cannot be verified from the supplied material.
- The authors themselves flag an inconclusive first test and analysis choices made after outcomes were known, so the external-validity result should be read cautiously.
- The benchmark is simulated around one bank contact center setting; real-world performance beyond the 33 real changes cited is not established.
For security and IT teams
- Scope
- The benchmark covers simulated organizational decisions in a bank contact center plus 33 real organizational changes used for an external check; it is not a general-purpose forecasting tool.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.