← News

Models & toolsSources: arXiv · 18 Nov 2026

GIB-200K benchmark shows vision-language models misweight text versus image evidence

A new arXiv paper introduces GIB-200K, a stress-test dataset measuring how retrieval-augmented vision-language models balance retrieved text against image evidence, finding large accuracy drops and modality biases.

In plain words

Some AI systems answer questions by combining an image with retrieved documents. Researchers built a large test set that deliberately corrupts one or both inputs to see which source the models trust. The tested models leaned unevenly toward text or image, and accuracy dropped sharply under stress.

What was announced

The paper, a cross-listed arXiv abstract, reports that the authors built GIB-200K from COCO and Visual Genome images, expanding 200,000 image-question items with controlled stress conditions: hard-negative retrieval noise of one to twenty documents, visual ambiguity such as blur and low-resource crops, and cross-modal conflict using counterfactual flip pairs. Drawing on ideal Bayesian cue combination, the authors define Knowledge Inertia, where text is over-weighted relative to the model's own image evidence, and Sensory Capture, the reverse, and estimate the weights with a measurement-error-corrected regression they verify against known values.

On a frozen, seeded evaluation split of 2,000 items, the four open models tested scored 92.5% to 94.2% on clean inputs, but accuracy fell to 41.3% to 59.9% under adverse conditions, according to the abstract. The two LLaVA models over-weighted text, with weight ratios of 1.229 and 1.322, while InternVL3 favoured the image at 0.680. Between 34.8% and 71.4% of correct clean answers were abandoned when the retrieved document lied. A training-free consistency check comparing answers with and without retrieval traded coverage for accuracy under misleading documents, and falling back to the image failed when the image was heavily degraded. The authors release per-instance outputs, code, and an audit recomputing every reported open-model number.

What it does not show

  • The evidence is the paper's abstract only; full methodology, results, and peer review status are not verifiable from this excerpt.
  • Only four open models on one frozen 2,000-item split were evaluated, so findings may not extend to other or closed models.
  • The abstract reports no independent replication; the released audit is the authors' own recomputation.

For security and IT teams

Scope
The dataset, code, per-instance outputs, and a recomputation audit for the open-model results are released, per the abstract.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.