← News

Agents & MCPSources: arXiv · 18 Nov 2026

ForecastGym benchmarks forecasting agents on frozen web evidence

An arXiv paper introduces a benchmark that records search results for prediction questions and replays them offline, separating how well models reason from what evidence they retrieve.

In plain words

AI agents that predict real-world events usually search the web themselves, which makes it hard to tell whether good predictions come from a strong model or lucky search results. ForecastGym solves this by saving the search results collected every two days for prediction-market questions, then replaying that fixed evidence to different models after outcomes are known. The paper reports that the evidence a model sees can matter as much as the model itself.

What was announced

Researchers introduce ForecastGym, a benchmark for evaluating LLM-based forecasting agents under controlled evidence. The system records what four commercial web search APIs return for open prediction-market questions every two days, then replays this frozen evidence offline once outcomes are known. The first release covers 150 questions, 614 evidence states, and 131k search results, letting the authors compare models, memory strategies, and search sources under a fixed evidence-acquisition policy.

The design addresses a real evaluation problem: when agents search live on their own, retrieval mixes with reasoning, and retrospective searches can leak later information. The authors report that rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of questions, despite tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy, and diagnostic analysis suggests missed public information contributes more to large gaps than misreading does. Switching the search API changed a model's Brier score by up to 0.043, more than the spread across models, while persistent notes scored similarly to rereading all evidence at half the input tokens.

What it does not show

  • The evidence is limited to the paper's abstract, so methodological details and full results are not covered.
  • All findings are the authors' own claims from a single arXiv paper and have not been independently confirmed.
  • The benchmark covers prediction-market questions and four commercial search APIs, so results may not generalize to other forecasting domains.

For security and IT teams

Scope
ForecastGym's first release covers 150 prediction-market questions, 614 evidence states, and 131k search results from four commercial web search APIs.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.