← News

ResearchSources: arXiv · 16 Nov 2026

BigCodeArena adds live code execution to crowdsourced LLM evaluation

The arXiv paper describes an open human evaluation platform built on Chatbot Arena that lets reviewers run and interact with LLM-generated code before judging it.

In plain words

Judging AI-written code is hard because a reviewer must read long code and mentally simulate what it does. BigCodeArena is a platform where people can actually run the code an AI model produced and interact with the result before rating it. The team says this execution-based setup makes human preferences about coding models more reliable.

What was announced

The BigCodeArena authors report collecting more than 14,000 code-centric conversation sessions covering 10 widely used LLMs, 10 languages, and 8 types of execution environments, from which they identified over 4,700 multi-turn samples with pairwise human preferences. They say the data revealed underexplored model preferences in fine-grained domains defined by tasks, languages, and frameworks.

From this dataset the team curated two benchmarks. BigCodeReward, built from the post-processed conversations, measured how consistently reward models match human preferences, and the evaluation showed most LLMs judge coding preferences better when execution results are available. On that basis they propose AutoCodeArena, an automatic Elo rating benchmark intended to assess coding quality without human involvement, and report that proprietary models such as GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead recent emerging models in code generation.

What it does not show

  • The evidence is the paper's abstract only; the full article's methods, datasets, and detailed results are not covered here.
  • The reported figures and benchmark findings come from the authors themselves and are not independently confirmed.
  • The abstract does not describe platform availability, access conditions, or how the collected data is distributed.

For security and IT teams

Scope
The available source is the paper abstract, so platform details beyond what the abstract states are not covered.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.