← News

ResearchSources: arXiv · 16 Nov 2026

LEGAL-UQA benchmark tests legal question answering for Urdu speakers

Researchers built a parallel English-Urdu benchmark of 619 question-answer pairs from Pakistan's Constitution, and report that evaluation choices, retrieval baselines and human review all matter.

In plain words

A benchmark is a set of test questions with known answers used to check how well AI systems perform. LEGAL-UQA provides 619 paired English-Urdu questions and answers tied to passages of Pakistan's Constitution, so developers can measure whether software really helps Urdu speakers understand their rights.

What was announced

The authors evaluate two tasks separately: answering an Urdu question when the relevant source passage is supplied, and retrieving that passage from the document. They report that model rankings change depending on how answer similarity is measured, and that a simple word-matching method retrieved source passages more effectively than the embedding methods they tested.

The team also compared JEV, a decision model used as an automated judge, against human grades, finding that at the evaluated cutoff it agreed with humans less often than a baseline giving every answer full credit. An audit further identified cases where generated reference answers omitted information supported by the source passage. The authors conclude that legal QA systems for Urdu speakers need separate testing of retrieval and answering, simple retrieval baselines, and human validation of automated judges and reference answers.

What it does not show

  • The source is the paper's abstract only; full methods, model identities and detailed results are not available in the supplied passages.
  • Findings such as the JEV comparison apply only at the evaluated cutoff described by the authors and are not independently confirmed.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.