Study finds higher benchmark scores did not translate into more merged pull requests
A natural experiment during a 19-day access ban to Claude Fable 5 found no statistically significant productivity gain from using the higher-scoring model, the paper's authors report.
In plain words
Coding models are usually judged by benchmark scores, but this study asked whether a higher-scoring model actually helps developers ship more work. When a U.S. directive briefly blocked access to Claude Fable 5, users switched to a lower-scoring model, and their pull request merge rates barely changed. The authors conclude the score gap did not show up as measurable productivity.
What was announced
The arXiv paper describes an unplanned experiment: on June 12, 2026, a U.S. export control directive cut access to Claude Fable 5 for 19 days, forcing its users onto other models, mostly Opus 4.8, which scores 11 points lower on SWE-Bench Pro. Because the people and projects stayed the same while only the model changed, the authors treated the suspension as a natural experiment to isolate the model's effect on productivity.
They tracked GitHub pull requests from 297 Fable 5 users before, during, and after the ban, using 8,908 developers who stayed on Opus 4.8 as a control group, and repeated the comparison across five model upgrades involving 1,208 developers and 15,442 pull requests. In a difference-in-differences model with developer and date fixed effects, access to Fable 5 raised the 7-day merge rate by only 1.4 percentage points, an interval spanning -1.4 to +4.3 that was not statistically significant. The five upgrade comparisons showed a weighted mean of -0.9 percentage points, with all estimates negative and intervals including zero, and merged pull requests per week rose a non-significant 2.4%.
What it does not show
- The findings come from an abstract only; the full paper's methods and data were not available for review.
- The study covers Fable 5 users and five 2026 model upgrades, so results may not extend to other models, benchmarks, or time periods.
- The confidence intervals include zero, so the evidence does not establish that the models have equal productivity, only that no significant difference was found.
For security and IT teams
- Scope
- The evidence covers merge rates and merged pull request counts for GitHub users of Claude Fable 5 and Opus 4.8 during a June 2026 suspension, not general code quality or other work outcomes.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.