Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

arXiv:2609.16006v1 Announce Type: new Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what a model knows rather than how it behaves when giving open-ended recommendations, opinions, and guidance. We introduce AraBehave: 1,623 culturally grounded, open-ended Arabic prompts with 29,214 cultural-appropriateness judgments from native speakers across several Arab regions, plus a scoring model whose predictions correlate strongly with human judgments on unseen systems (Pearson r=0.74). Evaluating three Arabic-centric and three frontier LLMs, we find that cultural appropriateness is not a single capability but decomposes into two largely independent components: normative stance and grounded cultural accuracy. The best general-purpose and best Arabic-centric models score identically (3.84 vs. 3.83 of 5) yet almost never fail for the same reason: general-purpose models exhibit strong factual grounding but a culturally inappropriate normative stance, being penalized for secular framing and false balance on culturally settled matters (28--33% of their low-score rationales), while the best Arabic-centric model adopts the expected stance but is penalized for fabricated hadith and misquoted verses (29%). Stance is cheap and fragile: one sentence of cultural instruction lifts Gemini to 4.57, above every Arabic-specialized model. Conversely, a generic ``answer clearly and objectively'' prompt costs Allam-7B 0.68 points, while asking the same questions in English lowers scores for eve

Study evaluates cultural appropriateness of LLMs

This study highlights that large language models have nuanced cultural appropriateness issues with Arabic language and culture, revealing risks of misalignment and inappropriate normative stances. However, improvements are feasible via targeted cultural instruction, suggesting potential for more respectful AI interactions. The impact is material but not extreme.

This is a methodological contribution highlighting a real gap between safety benchmarks and cultural appropriateness, useful for improving AI deployment across cultures. It is neutral-to-mildly-positive: it surfaces a problem but also shows tractable fixes (instruction tuning, alignment data).

The paper presents a new dataset and method for evaluating cultural appropriateness of LLMs. The results show that cultural appropriateness is complex and that current models struggle with it. The findings suggest that cultural instruction can improve model performance, but the overall implications for human flourishing are uncertain.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   6ef1fee4-98ea-4f22-bb6b-d3c00c6250ef
           https://arxiv.org/abs/2609.16006
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      +2  (Mixed or uncertain)
   ↓
CONFIDENCE 75%