Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education

arXiv:2610.09460v1 Announce Type: new Abstract: As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.

Study finds limitations in LLM-generated grading and feedback.

This is a rigorous empirical study on a real dataset showing LLM grading tools have specific, measurable diagnostic shortcomings versus human instructors. The implication is materially concerning for uncritical adoption in education but it is narrow in scope and includes constructive guidance, not a broad societal claim.

LLMs in higher education grading appear to offer thorough analytical evaluation but systematically fail to deliver. They provide redundant diagnostic information, rarely detect misconceptions, and lack appropriate tone modulation. This can misrepresent student performance and hinder effective learning.

LLMs provide redundant and ineffective feedback. This undermines educational quality. Human oversight is still required.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   e85b4162-f8fc-419e-b038-3818e703d0ff
           https://arxiv.org/abs/2610.09460
   ↓
EVIDENCE   3 extracted excerpts
   ↓
MODEL RUN  4 runs, methodology 1.0.1
   ↓
SCORE      -50  (Adverse)
   ↓
CONFIDENCE 81%