BigCodeBench tests LLMs on complex, tool-heavy coding tasks
A new benchmark from the BigCodeBench paper asks language models to combine function calls from many libraries, and current models still trail human programmers by a wide margin.
In plain words
Researchers introduced BigCodeBench, a test set that checks whether large language models can write Python code that calls many different library functions to solve practical tasks. Across 1,140 tasks, the best models scored up to 60 percent, while humans reached 97 percent.
What was announced
The benchmark's authors argue that existing evaluations mostly cover short, self-contained algorithmic problems or single function calls, which does not reflect practical work like data analysis or web development. BigCodeBench instead challenges models to invoke functions drawn from 139 libraries across 7 domains, spanning 1,140 fine-grained tasks. Each task carries an average of 5.6 test cases with 99 percent branch coverage, a measure of how much of the code's branching logic the tests exercise. A companion variant, BigCodeBench-Instruct, rewrites the original docstrings into short instructions containing only essential information, testing whether models can follow compressed natural-language directions.
In an evaluation of 60 LLMs, the authors report that models are not yet able to follow complex instructions and use function calls precisely, with scores up to 60 percent compared with human performance of 97 percent. The paper presents this gap as evidence that further advancement is needed in compositional reasoning over tools. The findings come from the benchmark's own test suite, so they measure performance on these specific tasks rather than general programming ability.
What it does not show
- The source is the paper's abstract only; full methodology, model list details and error analysis are not available in the supplied excerpt.
- Scores are reported as 'up to 60%' without per-model breakdowns, so the performance of individual models cannot be determined from this evidence.
- The 97% human figure reflects performance on this benchmark's tasks and does not establish broader comparisons between humans and models.
For security and IT teams
- Scope
- The benchmark covers 1,140 tasks using function calls from 139 libraries across 7 domains, with BigCodeBench-Instruct as a natural-language-instruction variant.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.