Solver2System dataset tests whether LLMs understand the physics behind simulation code
A new arXiv paper introduces Solver2System, a dataset of PDE solvers in multiple representations designed to probe whether large language models recover the physical system behind code.
In plain words
Researchers built a dataset of physics simulation code, expressed as programs, equations, descriptions and numerical outputs, to test how well large language models understand the physical system rather than just the text. The models were checked on whether they stay consistent across these forms and whether they spot code that runs but models the physics incorrectly.
What was announced
The authors, announcing the work on arXiv, argue that scientific simulation code is a useful testbed for LLM scientific reasoning because a single system can be represented as code, equations, natural language and numerical trajectories, and because code can execute successfully while implementing invalid physics. Solver2System contains PDE solvers with multiple representations, controlled corruptions and lexical perturbations, and evaluates whether models can recover the underlying physical system through cross-representation consistency and through static and agentic detection of physical invalidity.
Across eight open-weight models, the study reports that LLMs systematically favor some representations when evidence conflicts, and that lexical perturbations to code alter how the models interpret other, unchanged representations. Execution improves physical-validity judgments, but weaker agents can judge correctly without identifying the causal defect. The authors conclude that LLMs may reach correct judgments despite inconsistent representation reconciliation or causal misdiagnosis.
What it does not show
- The evidence is the paper's abstract only; full experimental details, model identities and result magnitudes are not available in the supplied passages.
- The eight models are open-weight; no closed or proprietary model results are reported in the supplied evidence, and no independent replication is described.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.