SynthDCB benchmark uses world distillation to test device-control agents at scale
The arXiv paper SynthDCB describes a benchmark for environment awareness and device control built through world distillation, covering Home, Car, and Phone settings with thousands of generated tasks.
In plain words
A research team has built a benchmark that checks how well AI agents can understand and operate devices such as those in homes, cars, and phones. Their method, called world distillation, turns knowledge of how devices behave into executable components that can automatically generate test tasks. The result is a large set of verifiable tasks for measuring agent performance.
What was announced
According to the paper, SynthDCB instantiates Home, Car, and Phone environments with 114 device classes, 1,166 functions, and 95 events, organized into five task families with 32 subtypes. Explicit states and tool-call traces support task validation and agent evaluation. The authors evaluated 22 models on 2,560 test tasks and report that the best model achieved 88.8% mean Pass@1, a measure of first-attempt success.
The authors also report coverage results: at the richest knowledge level, world distillation covered 22 of 23 properties, 19 of 29 functionalities, and 30 of 39 event requirements from a real RobotVacuum protocol, while 1,000 sampled tasks semantically covered 81.6% of 854 reference instructions in a real-world corpus. These construction experiments are presented as evidence of token efficiency and expanded task coverage, though the figures come from the authors' own experiments and have not been independently confirmed.
What it does not show
- The evidence is limited to the paper's abstract; full methodology, experimental details, and caveats are not available in the supplied passages.
- Reported results, including the 88.8% Pass@1 score and coverage figures, are the authors' own measurements and have not been independently verified.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.