Dutch word-puzzle benchmark tests how well language models handle ambiguity
The Shared Word (TSW-NL) asks solvers to find one Dutch word behind three ambiguous clues. Its creators report humans outscore most open-weight models.
In plain words
Researchers built a Dutch puzzle benchmark where each item gives three clues that all point to different senses of the same word. Solvers must combine the clues to guess the word. The authors say it fills a gap left by translated, multiple-choice tests that are hard to score fairly.
What was announced
The benchmark, described in an arXiv paper, is built from Dutch Wikipedia disambiguation pages, then filtered automatically and curated by hand. The authors list four design principles: the task is specific to Dutch, accepts open-vocabulary answers rather than fixed choices, is curated by humans, and is scored deterministically. The released dataset contains 222 instances, each with three independent human answers collected during testing.
In the authors' human study, people scored between 49.5% and 61.0%, while the best open-weight model they tested reached 39.3% and many scored under 20%. Closed models fared better: GPT-4.1 from April 2025 scored 59.2%, and GPT-5.6 Sol with maximum reasoning reached 88.8%. The team positions TSW-NL as a Dutch evaluation addition and a recipe that can be copied for other languages.
What it does not show
- The evidence is the paper's abstract only, so the scoring method, curation details and full results cannot be verified from the supplied passages.
- All scores come from the authors' own study; no independent evaluation is reported.
- The abstract does not name which open-weight models were tested or how many closed models were compared.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.