Who Can Evaluate AI Independently? A Framework for Analysing Evaluation Infrastructure

arXiv:2611.08134v1 Announce Type: new Abstract: AI evaluation is increasingly central to AI safety and governance, yet taking part in an evaluation does not by itself confer authority over its design, its execution or the resources it depends on. This paper examines how infrastructure arrangements distribute that authority, and so shape who can evaluate AI systems independently. Drawing on Science and Technology Studies, political economy and applied evaluation standards, we introduce PAAI (Power, Access, Adequacy, Independence), a framework for examining who controls evaluation infrastructure, the terms on which it is available, whether it is adequate to the evaluation's purpose and what autonomy it supports. We apply PAAI to twelve African AI evaluation projects (2022-2025) and to the documented roles of all ten member institutes in the three published joint testing exercises of the international network of AI Safety Institutes. We identify three recurring configurations: extractive collaboration, dependent innovation and constrained independence. African actors often contribute data, language expertise and task design while external institutions retain authority over infrastructure and evaluation frameworks; design authority without infrastructure authority never yielded more than partial independence. The joint testing exercises show a similar division among mostly high-income members: the institutes that led each exercise also ran the models, while Kenya, the network's only African member, moved from annotation to task design with no documented model execution. Partici

The paper highlights unequal distribution of authority in AI evaluation, limiting independence and control for African actors.

The paper discusses the distribution of authority in AI evaluation infrastructure, highlighting inequalities that can limit independent evaluation. This has ambiguous societal implications: it identifies governance challenges but does not assert direct positive or negative societal outcomes. The impact is contingent on how these issues are addressed.

This is a structural governance analysis with documented case studies showing that AI safety evaluation authority is concentrated in high-income institutions despite broader participation. The implication is materially concerning for equitable global AI oversight, though it concerns governance structure rather than AI capability itself.

The paper identifies a critical problem where participation in AI evaluation does not confer actual authority, particularly for African institutions. This unequal distribution of power over evaluation infrastructure and methods undermines the potential for independent AI safety and governance. It implies that current evaluation practices are structurally biased, hindering credible oversight and potentially perpetuating existing global inequalities.

The text describes a power imbalance in AI evaluation. Local actors contribute resources but lack control over infrastructure and evaluation frameworks. This limits their independence and the credibility of AI governance.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   db88d9c9-8e9a-46ab-b1cd-c27c7c8dff86
           https://arxiv.org/abs/2611.08134
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -42  (Adverse)
   ↓
CONFIDENCE 64%