Reforge benchmark ties LLM reverse-engineering scores to alignment confidence
An arXiv paper proposes a provenance-tracked pipeline for building function-level ground truth in decompiled binaries, arguing that unreliable source-to-binary alignment, not model capability, distorts current LLM evaluations.
In plain words
Researchers testing whether large language models can name functions in decompiled code need a trusted mapping between original source and binary output. The Reforge paper says that mapping is often unreliable, so accuracy numbers may be misleading. It proposes a pipeline that tracks provenance and flags low-confidence alignments.
What was announced
The authors argue that existing benchmarks for LLM-assisted binary analysis treat function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. They identify the main obstacle to fair evaluation as the reliability of binary-to-source alignment under compiler optimization, rather than model capability itself. Reforge addresses this by constructing ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, with an eight-gate confidence funnel and three-tier stratification to operationalize alignment uncertainty.
On a controlled micro-benchmark, the paper reports that high-confidence yield falls from 87.2% to 65.9% across optimization levels, and that unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming is described as demonstrating the validity of the concept. The abstract presents these results as motivation for uncertainty-aware evaluation; it does not report detailed model rankings, and the findings rest on the authors' own pipeline and controlled benchmark rather than independent replication.
What it does not show
- Evidence is limited to the paper's abstract; full methodology, model results and rankings are not available in the supplied passages.
- The yield figures and survivorship-bias finding come from the authors' own controlled micro-benchmark and have not been independently confirmed.
For security and IT teams
- Scope
- The paper concerns benchmarking methodology for LLM reverse-engineering of decompiled binary functions; the abstract does not describe a deployable tool or product.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.