CTF-Dojo trains language model agents on 658 executable capture-the-flag challenges
The arXiv paper introduces a large-scale Docker-based runtime of CTF-style challenges with verifiable feedback, plus an automated pipeline that builds new training environments from public artifacts.
In plain words
The authors built a large set of hands-on hacking-style challenges that AI agents can attempt inside containers, where success is automatically checkable. Training on the resulting verified attempts improved agent performance on three public benchmarks. The environments are open rather than proprietary, so the training signal does not depend on closed commercial systems.
What was announced
According to the paper's abstract, CTF-Dojo packages 658 fully functional Capture-The-Flag challenges in Docker containers with guaranteed reproducibility, addressing the scarcity of scalable, execution-grounded environments for training machine learning agents. The authors also describe CTF-Forge, an automated pipeline that converts publicly available artifacts into ready-to-use execution environments in minutes, replacing weeks of expert configuration. Training on 486 execution-verified trajectories from the runtime produced absolute gains of up to 9.9% over strong baselines on InterCode-CTF, NYU CTF Bench, and Cybench.
The abstract reports that the best-performing 32B model reached 31.9% Pass@1, which the authors describe as a new open-weight state of the art rivaling frontier models such as DeepSeek-V3-0324 and Gemini-2.5-Flash. They argue that execution-grounded training signals are pivotal for advancing high-performance agents without reliance on costly proprietary systems. These figures come from the authors' own evaluation and have not been independently confirmed.
What it does not show
- Evidence is limited to the paper's abstract; the full article was not reviewed, so methodology details and evaluation caveats are unverified.
- Benchmark results and the state-of-the-art claim are the authors' self-reported figures without independent confirmation.
For security and IT teams
- Scope
- The runtime targets CTF-style security challenges for agent training, not general production security testing.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.