Large language models (LLMs) are increasingly used to answer medical questions; however, their performance may vary depending on task characteristics. This study evaluated the performance of multiple versions of two widely used LLM families on oral and maxillofacial radiology (OMFR) questions from the Turkish Dental Specialty Examination (DUS) across three assessment phases and examined the influence of cognitive complexity, model family, evaluation phase, and content domain on response accuracy. A comparative repeated-evaluation design was used. A total of 123 text-based OMFR questions from DUS examinations (2012–2021) were submitted to two widely used LLM families (ChatGPT and DeepSeek) across three evaluation phases (May 2025, August 2025, and February 2026). Questions were categorized by content domain and Bloom cognitive level (low vs. high). Model responses were evaluated using official answer keys, and generalized estimating equations (GEE) were applied to account for repeated measurements. A total of 1230 model responses were analyzed, yielding an overall accuracy of 83.7%. Agreement between repeated runs was substantial to almost perfect (κ = 0.689–0.912). Cognitive complexity emerged as the strongest determinant of performance, with low-level questions significantly more likely to be answered correctly than high-level questions (OR = 6.15, p = 0.003). Content domain was also associated with accuracy (p = 0.028), whereas no statistically significant associations were observed for model family or evaluation phase. LLMs demonstrated high accuracy in answering OMFR examination questions; however, performance was more strongly associated with cognitive complexity than with model family or evaluation phase.
Department of Dentomaxillofacial Radiology, Faculty of Dentistry, Afyonkarahisar Health Sciences University, 03030, Afyonkarahisar, Turkey
This study did not involve human participants, patient data, or animals; therefore, ethics committee approval and informed consent were not required.
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.