← News

Agents & MCPSources: arXiv · 20 Nov 2026

Researchers open-source pipeline for validating agentic LLM serving before deployment

An arXiv paper describes an end-to-end evaluation pipeline for agentic LLM serving, showing trace replay reproduces real session loads far more closely than live re-runs.

In plain words

Companies increasingly run AI agents on their own open-weight models instead of paid APIs, and need to test serving systems before production. The authors built a pipeline that replays recorded agent sessions to recreate realistic load. Their results suggest this replay method is far more repeatable than re-running agents live.

What was announced

The paper, an arXiv abstract, argues that as enterprises move agentic workloads from closed APIs to self-hosted open-weight models, validation across the whole lifecycle matters, yet rollouts are usually checked with ad hoc tools whose accuracy is rarely verified. The authors present an end-to-end evaluation pipeline for agentic LLM serving and ground its design choices empirically. Replaying captured traces reproduced the per-session load of original agent sessions: at the 95th percentile, replay stayed within 1.18x of recorded input tokens, while live re-runs reached 4.98x, meaning a small set of recorded sessions yields a repeatable token load.

A code-coverage analysis of vLLM found that live, replayed and synthetic workloads were complementary in the tested configuration, each exercising Python serving paths the others did not reach. In a case study rolling out four large mixture-of-experts models on llm-d, the target models reached roughly half the optimal concurrency, and about 60% of the throughput, on captured workloads compared with public traces. The authors open-source the tools and release a trace dataset.

What it does not show

  • The evidence is the paper's abstract only; the full article's methods, configurations and caveats are not covered here.
  • Reported figures, including the 1.18x and 4.98x comparisons and the llm-d case-study results, come from the authors' own experiments and have not been independently confirmed.
  • The code-coverage complementarity finding is stated for the evaluated configuration and may not generalize to other setups.

For security and IT teams

Scope
The released tools and trace dataset target validation of agentic LLM serving platforms such as vLLM and llm-d, per the authors' abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.