PILAPS benchmark finds LLM pragmatism varies across English and Mandarin
A new bilingual benchmark tests whether large language models' pragmatic skills hold up across languages and conversational settings, and the authors report that strong English results do not reliably transfer.
In plain words
Pragmatics means understanding what someone really means beyond the literal words, such as catching a joke or an implied request. Researchers built PILAPS, a test set in English and Mandarin Chinese covering eight such phenomena. They found that models good at this in one language or setting often do worse in another.
What was announced
The benchmark pairs purpose-written items with dialogue-derived ones, and includes both culturally adapted and independently sourced data. The authors evaluated eight models from four families across different sizes and prompting strategies, so the comparison spans both open and closed systems rather than a single model type.
Results varied substantially across models and phenomena. Open-source models struggled most with humour, and cross-linguistic performance tracked model origin. Compared with human participants, models more often gave literal interpretations in conversational contexts. Theory-of-Mind prompting reduced that literal share of errors, while Chain-of-Thought prompting slightly increased it. The authors conclude that strong pragmatic performance in one evaluation setting does not necessarily generalise across languages or conversational contexts.
What it does not show
- The evidence is the paper's abstract only; full methodology, item counts, and detailed scores are not available in the supplied passages.
- Findings come from the benchmark authors' own evaluation and have not been independently confirmed.
- The abstract does not name the eight models or four families, so specific model-level results cannot be reported.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.