Situating Vulnerability: An Ethical Governance Framework for Anthropomorphic AI Interaction

arXiv:2610.29304v1 Announce Type: new Abstract: Large language models increasingly address users in the register of care: they remember personal details, adopt a first-person emotional voice, and sustain relationships across many sessions. This anthropomorphic mode of interaction generates risks that output-level metrics capture poorly, because the same empathic utterance may console one user, entrap a second, and consolidate a delusion in a third. Governance has responded in two ways. Uniform restriction refuses sensitive requests or suppresses empathic phrasing across all outputs; group-targeted protection profiles vulnerable populations and applies stricter or prohibitive rules to them. This article argues that both fail. Uniform restriction mislocates harm in the lexical form of responses rather than in the trajectory of a relationship; group-based protection depends on a list that can never be complete, treats a fluid property as a static attribute of persons, and converts protective attention into stigmatizing labels. Drawing on philosophical work that treats vulnerability as universal, relational, and differentiated, we propose that the unit of ethical governance should be the vulnerability situation rather than the vulnerable group, develop a typology of five situations - physical, emotional, cognitive, autonomy, and relational - and specify seven principles into requirements that guide response calibration without diagnosing or persistently classifying users. The framework relocates AI safety from output control to relational safety.

The article highlights risks of anthropomorphic AI interaction, including entrapment and consolidation of delusions, and argues that current governance approaches are inadequate.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   4ba382a4-2d75-4221-b48c-7e468fc1098d
           https://arxiv.org/abs/2610.29304
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  1 run, methodology 1.0.1
   ↓
SCORE      +50  (Favourable)
   ↓
CONFIDENCE 80%