← News

ResearchSources: arXiv · 19 Nov 2026

LexTemporis grades LLM-written legal code against real Korean SME records, not a grader's reading

A new benchmark checks whether programs that apply legal rules match what actually happened, using disclosures from Korean listed companies. The authors argue grading against a reader's interpretation can overstate performance.

In plain words

Laws are increasingly turned into computer programs, often written by large language models, that decide cases such as whether a firm counted as a small or medium enterprise on a date. Usually such programs are checked against answers from a person who read the same law, so they can score well by copying that person's interpretation. LexTemporis instead checks programs against records of real outcomes.

What was announced

The benchmark, described in an arXiv abstract, uses 10,901 firm-years of SME status disclosed by Korean listed companies, paired with every version of the governing rules; appendix cases cover US income tax and German hospital minimum volumes. The authors report that dates change outcomes mainly through provisions outside the main statutory text, with one transitional provision raising agreement with the disclosures from 83-86% to 89-90%. The records also test the author's own reading of a contested provision, favouring it in 130 of 189 firm-years where readings diverge.

Four LLMs wrote the rules from different selections of the law, and a pre-registered comparison found no condition better across all four models. In exploratory analyses, the apparent benefit of supplying the right provisions shrank for three models when graded against the records rather than the author's reading, in one case to between a quarter and three-fifths of its size, and was several times larger per firm-year where the author's reading was wrong. The authors conclude that grading legal programs against the grader's own reading can overstate how well they work.

What it does not show

  • The evidence is an abstract only, so methods, data construction and full results are not available for verification.
  • The LLM comparison is described as pre-registered with no condition better across all four models, and the stronger findings come from exploratory analyses the authors themselves flag as partly shaped by how the grader's reading was constructed.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.