Beyond Accuracy: A Psychometric Benchmark and Stability Analysis of 15 Large Language Models on NEET-PG Medicine Questions (2021--2025)

Anupam Kumar Singh; Neeshu Singh; Vinay Kumar Verma; Prasanth Gutti

Article Details

Beyond Accuracy: A Psychometric Benchmark and Stability Analysis of 15 Large Language Models on NEET-PG Medicine Questions (2021--2025)

Authors:

Anupam Kumar Singh | Neeshu Singh | Vinay Kumar Verma | Prasanth Gutti

Publisher:

Journal of Contemporary Clinical Practice (JCCP)

Journal:

Journal of Contemporary Clinical Practice Vol.11 Issue 12, pp.783-789

Document Type:

Article

Publication Date:

12/2025

Document Language:

English

Document ID (Org No.):

2001613703

pISSN:

2457-7200

Keywords:

Abstract:

 Background: Large language models (LLMs) are increasingly utilized in medical education and assessment, yet evaluations often rely solely on raw accuracy without accounting for stochastic variability or item difficulty. This study benchmarks 15 contemporary LLMs on the National Eligibility-cum-Entrance Test for Postgraduate (NEET-PG) Medicine examination, assessing accuracy, run-to-run stability, and response bias using psychometric methods. Methods: We evaluated models from major families (GPT, Gemini, Claude, Llama, Deep Seek, Kimi) using NEET-PG Medicine recall questions (n=121) from 2021--2025. Each model attempted every question three times to quantify reproducibility. Performance was analyzed using generalized linear mixed-effects models (GLMM) to adjust for year and difficulty, and Rasch Item Response Theory (IRT) to estimate latent ability. We further assessed run-to-run stability and option-selection bias. Results: GLMM-adjusted probabilities of a correct response varied widely, with GPT 5.2 (0.95; 95% CI 0.93--0.97) and Gemini 3 Flash (0.94; 95% CI 0.91--0.96) emerging as the top performers. While Claude Opus 4.5 demonstrated high run-to-run stability (98% identical responses), Claude 3.5 Sonnet showed substantial variability (41%). Systematic option-choice bias was observed, particularly in Llama models which under-selected option A (-15.7%) and over-selected D (+8.8%). A majority-vote ensemble achieved 87.6% accuracy but did not surpass the single best model. Rasch analysis confirmed a strong concordance with GLMM rankings (Spearman rho = 1.00). Conclusion: Newer LLMs demonstrate high proficiency on NEET-PG Medicine items, but raw accuracy masks significant differences in reliability and response bias. High-performing models are generally more stable and calibrated, whereas others exhibit marked stochasticity and letter preferences. These findings emphasize that clinical and educational adoption of LLMs requires rigorous psychometric validation beyond simple pass rates.

Article Details

Beyond Accuracy: A Psychometric Benchmark and Stability Analysis of 15 Large Language Models on NEET-PG Medicine Questions (2021--2025)

Authors:

Anupam Kumar Singh | Neeshu Singh | Vinay Kumar Verma | Prasanth Gutti

Publisher:

Journal of Contemporary Clinical Practice (JCCP)

Journal:

Journal of Contemporary Clinical Practice Vol.11 Issue 12, pp.783-789

Document Type:

Article

Publication Date:

12/2025

Document Language:

English

Document ID (Org No.):

2001613703

pISSN:

2457-7200

biosafety

knowledge

attitude

and practice (kap)

senior secondary school students

cross-sectional study

Issues

Article Details

Beyond Accuracy: A Psychometric Benchmark and Stability Analysis of 15 Large Language Models on NEET-PG Medicine Questions (2021--2025)

Authors: Anupam Kumar Singh | Neeshu Singh | Vinay Kumar Verma | Prasanth Gutti

Publisher: Journal of Contemporary Clinical Practice (JCCP)

Journal: Journal of Contemporary Clinical Practice Vol.11 Issue 12, pp.783-789

Document Type: Article

Publication Date: 12/2025

Document Language: English

Document ID (Org No.): 2001613703

pISSN: 2457-7200

biosafety

knowledge

attitude

and practice (kap)

senior secondary school students

cross-sectional study

Issues

Authors:

Anupam Kumar Singh | Neeshu Singh | Vinay Kumar Verma | Prasanth Gutti

Publisher:

Journal of Contemporary Clinical Practice (JCCP)

Journal:

Journal of Contemporary Clinical Practice Vol.11 Issue 12, pp.783-789

Document Type:

Article

Publication Date:

12/2025

Document Language:

English

Document ID (Org No.):

2001613703

pISSN:

2457-7200