← News

Agents & MCPSources: arXiv · 16 Nov 2026

ShopGym framework turns live storefronts into reproducible test shops for e-commerce agents

Researchers introduce ShopGym, a simulation and benchmarking framework whose generated sandbox shops aim to combine the realism of live storefronts with controlled, repeatable evaluation.

In plain words

AI shopping agents are usually tested either on real store websites, which change constantly and make results hard to repeat, or on small hand-built test sites, which are easy to control but too limited. ShopGym is a new framework that automatically builds self-contained copies of real stores so agents can be tested under stable, inspectable conditions. The authors report that agents score similarly on these copies and on the live stores.

What was announced

The arXiv paper, from the ShopGym authors, argues that the field's core bottleneck is methodological: live storefronts are realistic but non-stationary and irreproducible, while hand-built sandboxes are controllable but cover only narrow layouts, catalogs, and policies. The framework has two layers. ShopArena converts live seed storefronts into self-contained sandbox shops using anonymized shop specifications and a staged, validated generation process. ShopGuru then synthesizes benchmark tasks across seven skill categories, each grounded in a shop's catalog, navigation structure, policies, and interaction affordances. The output is described as resettable, inspectable, and stable evaluation artifacts.

Validation, as reported in the abstract, used graph-based structural analysis and agent-based behavioral evaluation covering 224 generated tasks across six sandbox shops, three built on synthetic data and three on real data. The authors state that synthetic shops preserved key structural properties of live storefronts and that agent success rates on synthetic twins closely tracked live storefronts, with matching per-skill profiles and failure modes. Generating ten shops showed the pipeline scales at roughly $60 per shop with declining human effort. These figures come from the authors' own evaluation and have not been independently confirmed.

What it does not show

  • Evidence is limited to the paper's abstract; the full article's methods, datasets, and detailed results were not available for review.
  • All performance and cost figures are the authors' own reported results; no independent confirmation is included in the supplied material.

For security and IT teams

Scope
The framework targets evaluation of e-commerce web agents, producing sandbox shops and benchmark tasks rather than a deployed shopping system.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.