Study finds language models miss ambiguous data questions before writing code
A diagnostic study of twelve models on the BIRD-Pandas benchmark suggests the hardest part of natural-language data querying is spotting when a question needs clarification, not answering it.
In plain words
Language models can turn plain-English questions about structured data into runnable programs, but a question may allow several valid interpretations. The paper introduces a way to test whether models notice these unresolved choices before coding, or silently pick one and risk wrong results. The authors call that silent guessing behavior Underspecified Decision Completion.
What was announced
Researchers behind the arXiv paper introduce Grounded Code Generation, a diagnostic protocol that splits the task into separate stages: identifying unresolved decisions, resolving them with evidence, and translating them into executable code. Examining each stage in isolation lets the authors pinpoint where decisions that led to incorrect results were first missed or mishandled.
Testing the framework on BIRD-Pandas with twelve models, the authors report that answering clarification questions consistently improves execution accuracy, yet substantial errors remain. Their conclusion is that the main difficulty lies not in answering clarification questions but in recognizing which questions need to be asked in the first place. The findings are limited to the benchmark and models tested, and the reported results come from the authors' own evaluation rather than independent confirmation.
What it does not show
- The evidence is the paper's abstract only, so the full methodology, per-model results and error analysis are not available to verify.
- Results are reported by the paper's authors on a single benchmark (BIRD-Pandas) with twelve models; no independent replication is described.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.