GENERATIVE ARTIFICIAL INTELLIGENCE AND THE VALIDITY CRISIS OF ASSESSMENT IN MEDICAL EDUCATION: AN INTEGRATIVE REVIEW
DOI:
https://doi.org/10.18623/rvd.v23.8513Keywords:
Generative Artificial Intelligence, Large Language Models, Medical Education, Educational Measurement, Validity, Academic IntegrityAbstract
Introduction: The rapid spread of generative artificial intelligence tools based on large language models has challenged the assumptions that support assessment in medical education. When a chatbot can answer licensing-style questions, write reflective narratives, generate test items, and score clinical performance, the interpretations drawn from assessment results may no longer correspond to the competencies they are intended to represent. Objective: To examine the scientific evidence on the effects of generative artificial intelligence on the validity of assessment in medical education, considering its roles as examinee, item writer, and assessor. Method: An integrative literature review was conducted, grounded in the principles of evidence-based practice. The review question was structured using the PICo strategy. The search was carried out in the PubMed/MEDLINE, Scopus, Web of Science, and ERIC databases. Included studies were characterized in terms of their methodological features and levels of evidence, and the findings were subjected to interpretive thematic synthesis. Results: Eleven studies were included in this review. The literature indicates that generative artificial intelligence performs at or above the level of students in knowledge-based and written tasks, that educators show limited ability to identify texts produced by these tools, that automatically generated items require expert review to preserve content validity and cognitive level, and that AI-based scoring shows limited agreement with human judgment and may introduce linguistic bias against specific groups of students. Conclusion: The validity crisis associated with generative artificial intelligence does not stem from the technology alone, but from assessment formats that depend on unsupervised written products and isolated measurements. Programmatic assessment, supervised performance-based methods, transparent institutional policies, and qualified human oversight of AI-supported processes are needed to preserve the credibility of assessment decisions.
References
AL-SHAWEE, N.; MCELVANEY, G.; STRAWBRIDGE, J.; SPOONER, M. When assessment theory meets generative AI: reimagining SBA design in medical education. Perspectives on Medical Education, v. 15, n. 1, p. 239-250, 2026. Available at: https://doi.org/10.5334/pme.2033. Accessed on: 18 Aug. 2026.
DORU, B.; MAIER, C.; BUSSE, J. S.; LÜCKE, T.; SCHÖNHOFF, J.; ENAX-KRUMOVA, E.; HESSLER, S.; BERGER, M.; TOKIC, M. Detecting artificial intelligence-generated versus human-written medical student essays: semirandomized controlled study. JMIR Medical Education, v. 11, e62779, 2025. Available at: https://doi.org/10.2196/62779. Accessed on: 20 Aug. 2026.
KANE, M. T. Validating the interpretations and uses of test scores. Journal of Educational Measurement, v. 50, n. 1, p. 1-73, 2013. Available at: https://doi.org/10.1111/jedm.12000. Accessed on: 15 Aug. 2026.
KIYAK, Y. S.; EMEKLI, E. ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgraduate Medical Journal, v. 100, n. 1189, p. 858-865, 2024. Available at: https://doi.org/10.1093/postmj/qgae065. Accessed on: 16 Aug. 2026.
KUNG, T. H.; CHEATHAM, M.; MEDENILLA, A. et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digital Health, v. 2, n. 2, e0000198, 2023. Available at: https://doi.org/10.1371/journal.pdig.0000198. Accessed on: 15 Aug. 2026.
LAW, A. K. K.; SO, J.; LUI, C. T.; CHOI, Y. F.; CHEUNG, K. H.; HUNG, K. K. C.; GRAHAM, C. A. AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination. BMC Medical Education, v. 25, e208, 2025. Available at: https://doi.org/10.1186/s12909-025-06796-6. Accessed on: 17 Aug. 2026.
LOPES, R. L.; KUBRUSLY, M.; ARRUDA, J. S.; PEIXOTO JUNIOR, A. A.; GUIMARÃES, M. M.; ROCHA, J. L. C.; ROCHA, H. A. L. Improving medical assessment validity with generative AI: lessons from human-AI interaction in Bloom's taxonomy classification. Medical Science Educator, v. 36, n. 3, p. 1341-1351, 2026. Available at: https://doi.org/10.1007/s40670-026-02719-2. Accessed on: 21 Aug. 2026.
MASTERS, K.; MACNEIL, H.; BENJAMIN, J.; CARVER, T.; NEMETHY, K.; VALANCI-AROESTY, S.; TAYLOR, D. C. M.; THOMA, B.; THESEN, T. Artificial intelligence in health professions education assessment: AMEE Guide No. 178. Medical Teacher, v. 47, n. 9, p. 1410-1424, 2025. Available at: https://doi.org/10.1080/0142159X.2024.2445037. Accessed on: 16 Aug. 2026.
MELNYK, B. M.; FINEOUT-OVERHOLT, E. Evidence-based practice in nursing & healthcare: a guide to best practice. 4th ed. Philadelphia: Wolters Kluwer, 2019.
NDACYAYISENGA, J. C. Y.; KIDEGA, C.; ACIRO CAN, L. The impact of generative artificial intelligence tools on assessment equity for non-native English-speaking medical students: a systematic review. BMC Medical Education, v. 26, e989, 2026. Available at: https://doi.org/10.1186/s12909-026-09303-7. Accessed on: 22 Aug. 2026.
PAGE, M. J.; MCKENZIE, J. E.; BOSSUYT, P. M. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, v. 372, n71, 2021. Available at: https://doi.org/10.1136/bmj.n71. Accessed on: 15 Aug. 2026.
STRONG, E.; DIGIAMMARINO, A.; WENG, Y.; KUMAR, A.; HOSAMANI, P.; HOM, J.; CHEN, J. H. Chatbot vs medical student performance on free-response clinical reasoning examinations. JAMA Internal Medicine, v. 183, n. 9, p. 1028-1030, 2023. Available at: https://doi.org/10.1001/jamainternmed.2023.2909. Accessed on: 17 Aug. 2026.
WHITTEMORE, R.; KNAFL, K. The integrative review: updated methodology. Journal of Advanced Nursing, v. 52, n. 5, p. 546-553, 2005. Available at: https://doi.org/10.1111/j.1365-2648.2005.03621.x. Accessed on: 15 Aug. 2026.
WRAITH, C.; CARNEGY, A.; BROWN, C.; BAPTISTA, A.; SAM, A. H. Can educators distinguish between medical student and generative AI-authored reflections? Medical Education, v. 59, n. 11, p. 1214-1221, 2025. Available at: https://doi.org/10.1111/medu.15750. Accessed on: 20 Aug. 2026.
YOKOSE, M.; HIROSAWA, T.; SAKAMOTO, T.; KAWAMURA, R.; SUZUKI, Y.; HARADA, Y.; SHIMIZU, T. The validity of generative artificial intelligence in evaluating medical students in objective structured clinical examination: experimental study. JMIR Formative Research, v. 9, e79465, 2025. Available at: https://doi.org/10.2196/79465. Accessed on: 19
Downloads
Published
How to Cite
Issue
Section
License
I (we) submit this article which is original and unpublished, of my (our) own authorship, to the evaluation of the Veredas do Direito Journal, and agree that the related copyrights will become exclusive property of the Journal, being prohibited any partial or total copy in any other part or other printed or online communication vehicle dissociated from the Veredas do Direito Journal, without the necessary and prior authorization that should be requested in writing to Editor in Chief. I (we) also declare that there is no conflict of interest between the articles theme, the author (s) and enterprises, institutions or individuals.
I (we) recognize that the Veredas do Direito Journal is licensed under a CREATIVE COMMONS LICENSE.
Licença Creative Commons Attribution 3.0
