Physiognomy Revisited: How Vision-Language Models Encode Sensitive Attributes Through Image Transformation

arXiv:2610.35910v1 Announce Type: cross Abstract: Emergent generative AI technologies, while remarkably beneficial across several tasks and domains, can suffer from unforeseen vulnerabilities and difficult-to-anticipate failure modes. Such failure modes can disparately affect vulnerable communities. This paper investigates two well-known vision language models (VLMs) and inspects how these models encode protected identities while generating images. We show that when VLMs are requested to transform human face images to reflect race identities (e.g., Hispanic, Black, White, or Asian) and sexual orientation (e.g., gay or straight), VLMs often comply with such requests. We demonstrate that compliance (and refusal) is non-deterministic, group-sensitive, and generated images are statistically distinguishable, often accentuating deep-seated stereotypes. We further show that representation biases introduced during image generation can re-emerge in textual reasoning and characterization tasks. Finally, we show that this worrisome finding is not limited to racial identity or sexual orientation, image generation can perpetuate bias around criminality, raising concerning possibilities of AI-assisted profiling.

Vision-language models encode sensitive attributes, perpetuating stereotypes and biases.

The paper presents evidence that vision-language models encode and reproduce biases about race, sexual orientation, and criminality in image generation and textual reasoning, which can perpetuate harmful stereotypes and enable AI-assisted profiling. This poses material risks for vulnerable groups and societal fairness. The evidence is direct and experimentally grounded, supporting a materially concerning impact.

This is primary empirical evidence that current VLMs reproduce and amplify stereotypes around race, sexuality, and criminality, with bias crossing from image to text domains. The profiling implication is concerning but described as a possibility, not a deployed harm, so it stays short of extreme.

The study demonstrates that advanced AI models can be prompted to generate images reflecting protected identities, often relying on and accentuating harmful stereotypes. This inherent bias can then influence textual reasoning, raising serious concerns about AI-assisted profiling and perpetuating societal prejudices against vulnerable communities. The findings are based on direct observation of VLM behavior.

The paper shows that VLMs can generate biased images that perpetuate stereotypes. This can harm vulnerable communities and lead to AI-assisted profiling.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   63ac8d7d-ae0f-4c5c-9f94-f1e330f77283
           https://arxiv.org/abs/2610.35910
   ↓
EVIDENCE   4 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -76  (Strongly adverse)
   ↓
CONFIDENCE 86%