← News

Models & toolsSources: arXiv · 19 Nov 2026

MuDAS dataset shows majority labels hide how Arabic sexism detectors err

A new arXiv paper introduces MuDAS, a multi-dialect Arabic sexism dataset that keeps individual annotator labels, and finds that models with similar headline scores make very different mistakes.

In plain words

Datasets for detecting sexist comments usually store only the majority vote of annotators, discarding cases where people disagreed. The MuDAS project keeps every annotator's label for YouTube comments in three Arabic dialects. Testing showed that models with nearly equal scores fail in different ways, especially on comments where annotators disagreed.

What was announced

The paper's authors built MuDAS from YouTube comments in Saudi, Egyptian and Moroccan Arabic, with each comment labelled by three native speakers of its dialect and the individual annotations preserved rather than collapsed into a majority vote. They used the dataset to evaluate Arabic encoder models fine-tuned with hard-label, soft-label and perspectivist objectives, alongside GPT-5.6 Terra and four Arabic-centric large language models.

Model errors were more frequent on comments where annotators disagreed. Despite close Macro-F1 scores, GPT-5.6 Terra and the fine-tuned encoders showed contrasting error patterns: the authors report that GPT-5.6 Terra over-classified gender-related content such as religious advice on modesty, even when given the annotators' definition of sexism, while the encoders missed implicit and prescriptive sexism. The Arabic-centric models over-predicted sexism and scored below the encoders. Disagreement-aware learning preserved Macro-F1, and soft labels improved alignment with annotators on contested comments at the expense of unanimous ones, but neither approach reliably identified contested comments. Performance was lower on Egyptian Arabic, and the paper says this is not fully explained by the test-set composition and dialectness measures it examined. The authors conclude that majority-label scores can hide error patterns that matter for content moderation.

What it does not show

  • The evidence is the paper's abstract only; full method details, exact scores and dataset availability are not covered by the excerpt.
  • All findings are the authors' own reported results and have not been independently confirmed.
  • The abstract does not state which specific encoder or Arabic-centric models were used, nor the size of the dataset.

For security and IT teams

Scope
The paper's stated relevance to content moderation concerns error patterns on contested comments, which majority-label evaluation can obscure; the abstract does not describe a deployed moderation system.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.