Being judged through the record: A Rosenhan-inspired test of psychiatric label sensitivity in large language models

arXiv:2610.35326v1 Announce Type: new Abstract: The integration of large language models (LLMs) into clinical decision-making raises unresolved questions regarding diagnostic bias. We investigate whether a documented psychiatric history alters LLM recommendations when the current clinical evidence is identical. Using a counterfactual experiment inspired by Rosenhan's 1973 field study, we factorially varied diagnostic-history condition (no prior diagnosis stated vs. prior schizophrenia diagnosis) and current-evidence condition (voices ceased vs. voices persisted) within a clinical vignette. Across five LLMs (GPT-5.6 Sol, Claude Sonnet 5, Gemini 3.6 Flash, Grok 4.6, and Mistral Large 2512), we generated 20,000 model responses across independent queries. When evidence supported discharge, adding a prior schizophrenia diagnosis increased recommendations for continued hospitalization nearly fivefold, from 4.7% to 22.3% (risk difference, 17.5 percentage points; 95% CI, 16.2-18.8). This shift occurred across all five models (risk-difference range, 10.4-34.4 percentage points). In the same condition, reference concordance decreased from 95.3% to 77.7%, while stated decision confidence increased. Across all model-by-evidence contrasts, confidence increases corresponded to Cohen's d values of 0.35-1.09. Signal-detection analyses showed that the prior diagnosis shifted the decision criterion toward hospitalization across all five models and reduced discrimination between the two current-evidence conditions in three of them. When current evidence supported continued hospitalization, th

LLMs show diagnostic bias, increasing hospitalization recommendations with prior schizophrenia diagnosis.

The study shows that LLMs used in clinical decisions exhibit diagnostic bias by overemphasizing prior psychiatric labels, potentially leading to unnecessary hospitalization. This systemic bias could worsen patient treatment outcomes and fuel stigma. While evidence is solid and concerns concrete, the overall societal impact is moderate due to current LLM integration limits.

This is a well-powered, controlled study with large sample size and consistent effects across five major LLMs, showing real diagnostic bias with clinical stakes. The implication is concerning for AI use in healthcare but is a study of a current limitation, not a deployed harm, and includes value as a bias-detection tool.

The study provides robust empirical evidence from a counterfactual experiment across multiple LLMs. It demonstrates a significant and consistent bias in AI recommendations, where a prior psychiatric label overrides current clinical evidence, leading to negative outcomes like increased hospitalization recommendations.

The study shows that LLMs exhibit significant diagnostic bias, leading to unfair treatment based on past diagnoses. This bias is consistent across multiple models and can result in inappropriate hospitalization recommendations.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   e2d0bc78-0c77-42b6-894d-6a3d6d8eaa1f
           https://arxiv.org/abs/2610.35326
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -63  (Adverse)
   ↓
CONFIDENCE 87%