← News

ResearchSources: arXiv · 18 Nov 2026

MM-ReportBench finds top models score 68.9 on multimodal report generation

A new arXiv benchmark tests whether vision-language models can turn tables into reports with accurate text and faithful charts, and results show substantial weaknesses.

In plain words

Researchers released MM-ReportBench, a test set for models that must write reports from spreadsheet-style tables, including both narrative text and charts. The benchmark's authors found that even the best of 20 tested models scored 68.9 overall, with many reports containing numerical errors.

What was announced

The benchmark, described in an arXiv paper, contains 386 tasks built on 185 real-world tables. Each task comes with an expert-curated reference report that has standardized chapter structures and verifiable key insight points, assembled through a pipeline combining multimodal report generation based on Monte Carlo Tree Search with human quality control. The authors say existing benchmarks fall short because of single-modality focus, simplistic metrics, and weak handling of novelty and visualization quality.

Evaluation uses two judges: a Text Judge Model checks structural completeness, verifies insights with numerical fact-checking against the source tables, and analyzes novelty to separate genuine analytical discovery from trivial paraphrasing, while a Visualization Judge Model scores data fidelity, expressiveness, aesthetics, and technical correctness; both also assess cross-modal consistency. Across 20 state-of-the-art models, the strongest reached only 68.9 overall, numerical hallucination affected 38.5% of reports, and genuinely novel insights were rare. Ablation studies showed single-judge evaluation inflates scores by 8-12 points versus the dual-judge setup.

What it does not show

  • The evidence is the paper's abstract only; full methodology, model list, and task details are not covered by the supplied passages.
  • Scores and error rates are the authors' own reported results and have not been independently confirmed.
  • The passages do not name the models tested or specify which model achieved the 68.9 score.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.