EJBench tests whether AI models can judge engineering designs without running simulations
The benchmark's authors report that even the strongest of ten frontier models reaches a score of 0.71, and that models show similar confidence on wrong answers as on correct ones.
In plain words
A research team built a test called EJBench that checks if AI systems can draw sound conclusions from a handful of past engineering results, the way an engineer would before committing to a design. The tasks come from a real battery cooling design study, and answers are checked against converged simulations rather than judged by another AI. The reported scores show large gaps between the best and smallest models tested.
What was announced
The authors position EJBench differently from benchmarks that measure whether an agent can drive simulation or CAD tools. Their test is software-agnostic: it asks what conclusion an engineer would reach from a few earlier results, without launching a new simulation. The tasks derive from a conjugate heat transfer campaign for battery cold plates and fall into three groups: pairing unlabelled test results with their operating points, selecting which stored runs to reveal before approving a design, and ranking new duties by peak cell temperature.
Scoring is anchored to converged simulation output rather than an LLM judge. Among ten frontier models, the strongest, GPT-6 Astra, reached 0.71, and the authors note that models expressed nearly identical confidence on failed and successful tasks. The team also fine-tuned Qwen3.5-9B using LoRA and reinforcement learning, rewarding answers the simulations confirmed; on the held-out test split the score moved from 0.071 to 0.106. The train, validation and test splits, the evaluation harness and a leaderboard are being released.
What it does not show
- The source is an abstract, so task details, methodology and results are reported only at a high level and cannot be independently verified from this excerpt.
- The reported scores come from the benchmark's own authors; no independent evaluation is cited.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.