Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses

arXiv:2609.27063v1 Announce Type: new Abstract: This paper presents a multi-source classroom study conducted during a 10-week quarter in data science courses at Drexel University. We first investigate the behaviors of student engagement with large language models (LLMs) using four surveys across three data science courses. Second, we evaluate the construct validity of multiple-choice questions (MCQs) generated by an LLM for in-lecture retrieval practice. Based on 378 authored questions (311 deployed, producing 7{,}888 student responses), we analyze whether the difficulty ratings assigned by an LLM match empirical item difficulty. Our study shows that student engagement with LLMs varied across courses and increased over the term. Although students expressed high satisfaction and reported saving considerable time, their perception of deep learning benefits declined, and many noted a tendency toward over-reliance. Regarding the difficulty ratings of LLM-generated MCQs, the Easy, Medium, and Hard labels correlated closely with its assigned Bloom's Taxonomy levels (Spearman $\rho=0.90$), reflecting an artifact of co-generation. However, neither metric predicted empirical item difficulty (difficulty label $\rho=0.06$; Bloom level $\rho=0.02$). The ratings reflect the structural formatting of a question rather than its underlying difficulty.

LLMs may hinder deep learning

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   45da53e3-0deb-449b-839e-2659a37b5ced
           https://arxiv.org/abs/2609.27063
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  1 run, methodology 1.0.1
   ↓
SCORE      -50  (Adverse)
   ↓
CONFIDENCE 70%