SarcasTraps test set shows sarcasm trips up Turkish hate speech detectors
A new Turkish benchmark finds that sarcastic hate speech lowers recall in fine-tuned classifiers and precision in zero-shot LLMs, according to the paper's authors.
In plain words
Researchers built a small Turkish test set of 479 texts, each labeled for both hate speech and sarcasm, to see whether indirect, mocking language confuses hate speech detection systems. It does: models trained to spot hateful wording and large language models used without task-specific training both perform worse when sarcasm is involved.
What was announced
The SarcasTraps authors argue that standard hate speech datasets lean toward explicit harmful language, while sarcasm can carry harmful intent through mock praise, exaggeration, humor, or apparent politeness. To probe this gap, they annotated 479 Turkish entries along hate speech and sarcasm dimensions and used the set strictly as an external test, meaning the models were never trained on it.
Classifiers were built with BERTurk and XLM-RoBERTa, trained on two existing Turkish hate speech datasets, and GPT-4.1 and Llama-3.3-70B were tested zero-shot, without task-specific examples. The reported pattern is a trade-off: fine-tuned encoder models lose recall on sarcastic hate, missing more of it, while the zero-shot LLMs keep higher recall but drop substantially in precision, flagging more non-hateful sarcastic text as hateful. The authors present this as quantitative evidence that both approaches struggle with sarcasm in Turkish.
What it does not show
- The evidence is the paper's abstract only; full methodology, per-model numbers, and statistical detail are not available in the supplied passages.
- Results cover Turkish and the four named models; no other languages or systems were evaluated, and no independent confirmation of the findings exists.
- The dataset will be shared only on request for academic research purposes, so external verification depends on obtaining it from the authors.
For security and IT teams
- Access
- SarcasTraps is not openly released; the authors state it will be shared upon request for academic research purposes.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.