Who Gets Counted? Large Language Models, Informative Refusal, and Downstream Bias in Name-Based Demographic Classification

arXiv:2611.06550v1 Announce Type: new Abstract: Researchers routinely infer gender, race, and ethnicity from names, yet the standard tools used for this task misclassify the groups that diversity research most needs to measure. Large language models (LLMs) have been proposed as a remedy because they label diverse names accurately. We ask whether that accuracy carries over to the estimates researchers report. Across 10,000 Chinese names, 19,998 North Carolina voters with self-reported race, and 300 political science authors with human reference codes, we compare three LLMs with seven traditional tools and link the voter sample to party registration and age. Selective labeling explains part of the LLMs' accuracy advantage. Claude Haiku labels half of the voters, and a traditional tool limited to its most confident half is more accurate. Hard labels from every tool understate the White--Black gap in Democratic registration by 36% to 65%, while BIRDiE estimation on BISG posteriors comes within 0.03 of it. Haiku's missing labels select on age, inflating the age gap by six years. Combined with BISG, LLM answers sharpen individual labels without a demonstrated gain in BIRDiE accuracy. Classifiers should be judged by the estimates they produce, including the consequences of missing labels

LLMs may perpetuate biases in demographic classification, understating gaps in registration and inflating age gaps.

The text discusses technical evaluation of demographic classification accuracy using LLMs versus traditional tools, noting issues with selective labeling and biases. There is no clear claim about broader societal impact of AI or direct implications for human flourishing. The study mainly identifies challenges in demographic inference accuracy and biases without projecting systemic effects or benefits on society.

This is a methodological study about a specific research-tool failure mode, not a broad societal claim about AI's trajectory. It has real but narrow implications for research integrity in demographic measurement, warranting a mildly negative score for AI's current reliability in this niche application.

LLMs, when used for name-based demographic classification, exhibit selective labeling that distorts downstream statistical estimates. This leads to significant misrepresentation of demographic gaps, such as the White-Black gap in Democratic registration and age disparities, even when individual name labels appear accurate. The selective labeling exacerbates existing biases in research that relies on such classification.

The study finds that large language models (LLMs) and traditional tools both misclassify demographic groups. This leads to biased estimates in research, which can have materially concerning implications for societal equity and fairness.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   c9635ea7-9a03-4391-8ffc-0c69241a474f
           https://arxiv.org/abs/2611.06550
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -41  (Adverse)
   ↓
CONFIDENCE 83%