Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models

arXiv:2610.40893v1 Announce Type: cross Abstract: Fairness benchmarks for large language models (LLMs) are overwhelmingly English-born and American in their social categories; translated versions carry those categories into societies where they do not apply, and almost none fix their hypotheses before the models are run. We present Qistas (Arabic qistas, "the just balance"), a protocol for evaluating how LLMs treat the social, confessional, and regional constituencies of a society, portable across countries and instantiated first for Iraq. In its pilot form it measures only what needs no verdict on contested truth: procedural symmetry across matched counterfactual pairs, differential refusal, stereotype reliance under ambiguity, and terminological framing of documented events. Every measurement is paired by construction and analysed with paired machinery (sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, Holm-corrected families), with a power analysis that states, before any data exist, the smallest effect each axis can detect after the design effect of template clustering. We release the authoring apparatus (a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook) and report its first field test: a semantic review of a 58-item seed bank in which 24 of 25 template groups failed at least one check. The defect patterns it exposed (evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced

Exposes general hazards of counterfactual fairness benchmarks in LLMs

The paper proposes a fairness evaluation protocol for LLMs focused on non-Western languages and social categories, addressing a significant gap in fairness benchmarks. While improving fairness can benefit society, the paper itself does not evaluate models or guarantee broader impacts yet, making societal implications currently limited but promising.

This is infrastructure-building research with no direct claim about AI's societal effects yet; it only describes a protocol and pilot quality check, not model outcomes. Potential long-term benefit for fairer evaluation is plausible but unproven and speculative at this stage.

This paper introduces a novel protocol to improve fairness evaluation of large language models by addressing existing biases in benchmarks. It aims to make fairness assessments more culturally relevant and methodologically rigorous, which is a positive development for responsible AI. While no models have been evaluated yet, the framework itself is a step towards mitigating potential harms from biased AI systems.

The paper presents a protocol for evaluating fairness in LLMs. It does not evaluate any specific model. The protocol aims to address fairness across diverse societies, which is neutral to positive. The pilot found many failures in the template groups, but this is about the protocol's sensitivity, not the models' fairness.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   8aaaf842-4271-4d98-b362-96479746b894
           https://arxiv.org/abs/2610.40893
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -28  (Adverse)
   ↓
CONFIDENCE 64%