← News

ResearchSources: arXiv · 16 Nov 2026

Reforge benchmark ties LLM reverse-engineering scores to alignment confidence

An arXiv paper proposes a provenance-tracked pipeline for building function-level ground truth in decompiled binaries, arguing that unreliable source-to-binary alignment, not model capability, distorts current LLM evaluations.

In plain words

Researchers testing whether large language models can name functions in decompiled code need a trusted mapping between original source and binary output. The Reforge paper says that mapping is often unreliable, so accuracy numbers may be misleading. It proposes a pipeline that tracks provenance and flags low-confidence alignments.

What was announced

The authors argue that existing benchmarks for LLM-assisted binary analysis treat function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. They identify the main obstacle to fair evaluation as the reliability of binary-to-source alignment under compiler optimization, rather than model capability itself. Reforge addresses this by constructing ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, with an eight-gate confidence funnel and three-tier stratification to operationalize alignment uncertainty.

On a controlled micro-benchmark, the paper reports that high-confidence yield falls from 87.2% to 65.9% across optimization levels, and that unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming is described as demonstrating the validity of the concept. The abstract presents these results as motivation for uncertainty-aware evaluation; it does not report detailed model rankings, and the findings rest on the authors' own pipeline and controlled benchmark rather than independent replication.

What it does not show

  • Evidence is limited to the paper's abstract; full methodology, model results and rankings are not available in the supplied passages.
  • The yield figures and survivorship-bias finding come from the authors' own controlled micro-benchmark and have not been independently confirmed.

For security and IT teams

Scope
The paper concerns benchmarking methodology for LLM reverse-engineering of decompiled binary functions; the abstract does not describe a deployable tool or product.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.