← News

Models & toolsSources: arXiv · 20 Nov 2026

FEM-Bench expansion shows LLM coding agents still struggle with self-verification

A 17-task finite element benchmark finds the best model passes 87.1% of single-pass implementation trials, while agentic setups improve results but reveal persistent consistency and verification weaknesses.

In plain words

Researchers tested how well large language models write code for finite element analysis, a simulation method used in engineering. The best model wrote correct code in most single attempts, but no setup reliably succeeded every time. The main problem: models' own checks often miss their mistakes.

What was announced

The arXiv paper extends FEM-Bench to 17 finite element tasks covering nonlinear and history-dependent mechanics and structural dynamics, evaluating 15 models on implementation and test generation without execution feedback. The best model succeeded in 87.1% of single-pass implementation trials, but rankings differed between tracks: the leading implementer reached 49.9% joint test success, while the leading tester reached 82.6%. For six models, the authors compared single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solved some tasks that never succeeded in single-pass trials, but gains varied across models and harnesses, and regressions occurred.

Claude Opus 5 with SimpleAgent reached 96.5% Pass@3, meaning at least one success in three attempts, but only 77.1% Pass^3, meaning all three attempts succeeded; no model-harness combination exceeded 78.8% Pass^3. Analyzing 249 failed agentic trajectories, the authors identified six categories of self-verification failure, most often inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%); in 28.5% of cases the tests shared an error with the implementation. An exploratory configuration with separate coder and tester agents achieved the highest observed scores, 97.6% Pass@3 and 88.8% Pass^3. The authors conclude that consistency across attempts and independent verification are the main obstacles to trusting LLM-generated scientific code.

What it does not show

  • The source is an abstract-level excerpt; full task definitions, model list, and methodology details are not available in the supplied passages.
  • Results come from a single benchmark in finite element analysis and may not generalize to other scientific programming domains.
  • The coder-tester configuration is described as exploratory, and the abstract does not report independent replication of any results.

For security and IT teams

Scope
Findings apply to finite element simulation tasks in mechanics and structural dynamics, not general software engineering.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.