← News

ResearchSources: arXiv · 16 Nov 2026

BBOWP-Bench tests whether LLMs can formulate black-box optimization problems

A new arXiv benchmark asks large language models to design both a search space and an optimization algorithm from a plain-language description of a black-box task.

In plain words

Some problems can only be tested by running them, not written as equations; these are called black-box optimization tasks. The authors built a benchmark that checks whether AI models can read such a task in plain language and choose both what variables to search over and which optimization method to use.

What was announced

Formulating an optimization problem strongly affects solution quality, but good formulations usually require substantial expertise, the paper's authors write. They note that existing benchmarks mostly cover problems whose objectives and constraints can be written as explicit mathematical expressions, while many practical problems are black-box tasks where only objective values are observable. The new setting, Black-Box Optimization Word Problems (BBOWP), requires a system to infer both a search space and an optimization algorithm from a natural-language description.

The BBOWP Benchmark Suite pairs each instance with a natural-language description, an executable evaluation environment, and a human-designed reference formulation, so both search-space design and algorithm selection can be scored. In tests across multiple black-box domains, the authors report that current LLMs vary their optimizer choice with the evaluation budget, but sometimes struggle with search-space design, particularly when identifying important variables and balancing their ranges on less informative or highly problem-specific descriptions. Code and dataset are on GitHub.

What it does not show

  • The evidence is the paper's abstract only; full experimental details, baselines and results are not available in the supplied passages.
  • Findings about LLM performance come from the authors' own evaluation and have not been independently confirmed.

For security and IT teams

Access
The authors state that the code and dataset are available at https://github.com/shiralab/bbowp-bench.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.