Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT

arXiv:2609.35953v1 Announce Type: new Abstract: Young people increasingly turn to General-Purpose Conversational Agents (GPCAs), such as ChatGPT, in moments of distress. We examine young adults' (ages 18-25) experiences using ChatGPT. We first collected 19,930 ChatGPT conversations and survey data from 158 young adults. We then selected five example conversations reflecting user distress. Finally, we asked ten clinicians to review those five conversations. We found distressed participants reported greater emotional engagement with ChatGPT and greater behavioral change from using it than their peers. When they turned to ChatGPT in moments of acute distress, ChatGPT was quick to give overly dramatic responses and excessive action-oriented suggestions. Clinicians endorsed ChatGPT's availability and much of its wording, but identified seven process failures, such as prematurely jumping to solutions. We translated clinicians' feedback into design guidelines following three stages: 1) asking about safety, 2) de-escalating intensity to restore emotional regulation, and 3) exploring concerns without agreeing with them.

Study finds ChatGPT's responses can be overly dramatic, but also shows potential for emotional support.

This is a methodologically grounded but small-scale study (5 conversations reviewed by clinicians) identifying real but moderate failures in ChatGPT's handling of youth distress. It's constructive rather than alarmist, pointing toward design fixes, tempering the negative implication.

The study provides evidence that young adults in distress are using ChatGPT, which then responds with dramatic, action-oriented, and premature solutions. Clinicians identified significant process failures. This indicates a concerning potential for AI to negatively impact the emotional well-being of a vulnerable demographic.

The study found that ChatGPT can engage emotionally with distressed users and prompt behavioral change. However, it also identified significant failures in ChatGPT's responses, leading to design guidelines for improvement. The overall implication is positive but with clear room for enhancement.

SOURCE     arXiv cs.AI (tier 1)
   ↓
DOCUMENT   542c9b87-2eb3-4f2a-b420-7ff869a380e5
           https://arxiv.org/abs/2609.35953
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      -41  (Adverse)
   ↓
CONFIDENCE 78%