Realtime-Bench finds frontier models struggle when the clock is running
A new benchmark tests whether real-time streaming models can act under strict time limits, and the results suggest most cannot yet do so reliably.
In plain words
A research team built Realtime-Bench, a set of dynamic tasks where acting too late means failing. In the hardest setting, where the simulated world moves forward while the model is still computing, the best model succeeded only 44% of the time. The authors say real-time interaction remains an unsolved challenge for current models.
What was announced
The benchmark's authors argue that existing evaluations of real-time streaming models emphasise conversational quality and streaming perception, while leaving untested whether models can act effectively under strict temporal constraints. Realtime-Bench addresses this with curated dynamic tasks in which delayed actions directly affect success. The suite measures performance in three modes: Wall Clock, where the environment advances according to the model's actual inference latency; Paused Clock, an idealised reference that removes inference delays; and Token Clock, which advances simulated time based on generated token counts to test temporal awareness independently of serving speed.
Under Wall Clock conditions, the best success rate reached only 44%, with the best Pass^3 score at 36%, which the authors present as evidence that real-time interaction remains difficult. Comparing Wall Clock with Paused Clock results, they conclude that temporal costs limit success in evolving environments. The study also reports that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. The authors suggest future work on adaptive reasoning with temporal awareness and continuous perception for real-time multimodal models.
What it does not show
- The evidence is the paper's abstract only; full methodology, task details and complete results are not available in the supplied excerpt.
- Success rates and the reasoning-effort finding come from the authors' own benchmark; no independent evaluation is included.
- The abstract does not name which models were tested or how many, so the scope of the 44% figure cannot be verified from the excerpt.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.