Researchers open-source pipeline for validating agentic LLM serving before deployment
An arXiv paper describes an end-to-end evaluation pipeline for agentic LLM serving, showing trace replay reproduces real session loads far more closely than live re-runs.
In plain words
Companies increasingly run AI agents on their own open-weight models instead of paid APIs, and need to test serving systems before production. The authors built a pipeline that replays recorded agent sessions to recreate realistic load. Their results suggest this replay method is far more repeatable than re-running agents live.
What was announced
The paper, an arXiv abstract, argues that as enterprises move agentic workloads from closed APIs to self-hosted open-weight models, validation across the whole lifecycle matters, yet rollouts are usually checked with ad hoc tools whose accuracy is rarely verified. The authors present an end-to-end evaluation pipeline for agentic LLM serving and ground its design choices empirically. Replaying captured traces reproduced the per-session load of original agent sessions: at the 95th percentile, replay stayed within 1.18x of recorded input tokens, while live re-runs reached 4.98x, meaning a small set of recorded sessions yields a repeatable token load.
A code-coverage analysis of vLLM found that live, replayed and synthetic workloads were complementary in the tested configuration, each exercising Python serving paths the others did not reach. In a case study rolling out four large mixture-of-experts models on llm-d, the target models reached roughly half the optimal concurrency, and about 60% of the throughput, on captured workloads compared with public traces. The authors open-source the tools and release a trace dataset.
What it does not show
- The evidence is the paper's abstract only; the full article's methods, configurations and caveats are not covered here.
- Reported figures, including the 1.18x and 4.98x comparisons and the llm-d case-study results, come from the authors' own experiments and have not been independently confirmed.
- The code-coverage complementarity finding is stated for the evaluated configuration and may not generalize to other setups.
For security and IT teams
- Scope
- The released tools and trace dataset target validation of agentic LLM serving platforms such as vLLM and llm-d, per the authors' abstract.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.