← News

ResearchSources: arXiv · 16 Nov 2026

PerfArena ranks LLMs by code latency, memory and energy

A new open-source benchmark measures not just whether AI-generated code works, but what it costs to run, ranking 13 models across seven programming languages.

In plain words

Large language models can write code that works, but working code can still be slow or wasteful. PerfArena is an open-source tool that runs each model's code and measures how fast it is, how much memory it uses, and how much energy it consumes. The results are published as public leaderboards so developers can pick a model for each language and metric.

What was announced

The PerfArena team, writing in an arXiv paper, argues that most code-generation benchmarks judge only correctness, while performance benchmarks tend to cover few languages and rarely measure energy use. Their system executes each model's solutions on a Unix-like host and reports correctness alongside latency, peak memory and energy consumption. Judging components, profilers and code-generation workflows are pluggable, so new measurement backends can be added without changing the pipeline.

The authors evaluated the arena on two competitions: CLBGComp, ten compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. The current dataset holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across seven languages. Across both competitions, the paper reports that got-6-astra is the only model no other model significantly beats on latency, energy or peak memory, solving 99.7% of LeetCodeComp and all of CLBGComp. All leaderboards and measured solutions are publicly available.

What it does not show

  • The source is the paper's abstract, so methodology details, model versions and how rankings were computed are not covered in the supplied evidence.
  • The reported results come from the paper's authors; no independent verification of the measurements or rankings is included.
  • The dataset covers two competitions and seven languages, so the rankings may not generalize to other workloads.

For security and IT teams

Scope
PerfArena's public leaderboards cover measured solutions across seven programming languages and two competitions, intended as a reference for choosing a model per language and metric.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.