AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy

arXiv:2610.31993v1 Announce Type: new Abstract: AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.

AI models inform clinical decisions without transparency, and high error rates are found in benchmark tests.

The benchmark reveals that AI models vary widely in accuracy on Australian clinical policy and can confidently give incorrect advice, posing moderate risks in clinical decisions. However, identifying these issues helps improve safety and tailoring of AI in healthcare, offering potential benefits. The score reflects mixed but slightly concerning implications due to the high error rates and inconsistencies.

This is solid primary evidence of a real, measurable gap between model recall and actual current clinical policy, already deployed in practice without disclosure. The finding is concerning but narrow in scope, domain-specific, and the authors propose a constructive mitigation path, keeping this out of extreme territory.

The text describes a benchmarking study of AI models in clinical settings, showing high accuracy and potential for improvement. This implies a positive impact on healthcare outcomes and decision-making.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   ac95f867-f7a5-47e9-a7dd-fcc550b8189c
           https://arxiv.org/abs/2610.31993
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -21  (Adverse)
   ↓
CONFIDENCE 63%