JMIR MEDICAL EDUCATION LAUNCHES SPECIAL ISSUE ON THE USE OF CHATGPT IN MEDICAL EDUCATION, AFTER NEW STUDY FINDS CHATGPT PASSES THE UNITED STATES MEDICAL LICENSING EXAMINATION

News provided by

Mar 06, 2023, 07:30 ET

"The study found that ChatGPT reaches the equivalent of a passing score for a third-year medical student"

TORONTO and PHILADELPHIA, March 6, 2023 /PRNewswire/ -- A study published on February 8, 2023, in JMIR Medical Education, a leading open access journal on digital medical education, evaluated the potential of ChatGPT, a natural language processing model, as a medical education tool. The study found that ChatGPT reaches the equivalent of a passing score for a third-year medical student. Conducted by researchers from Yale University School of Medicine's Section for Biomedical Informatics and Data Science and University College Dublin, the study aimed to test the performance of ChatGPT and previous-generation large language models on the medical question-answering problem as part of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams.

In their paper titled "How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment," [1], Aidan Gilson and coauthors tested the models on study aids commonly used by medical students, including multiple-choice questions (with indicators of question difficulty) and the National Board of Medical Examiners (NBME) test sample questions. ChatGPT outperformed previous-generation models and was capable of correctly answering up to over 60% of questions, which is comparable to a passing score for a third-year medical student. Incorrect answers were primary due to logical and information errors, and the performance decreased as question difficulty increased. Impressively, ChatGPT provided logical reasoning and information internal to the question in most of its answer selection. Additionally, ChatGPT's responses provided external information beyond the question, which was significantly correlated with the performance. ChatGPT can provide a basis for dialogic interaction that is likened to studying with a peer, not only by giving a narrative, coherent answer but also by establishing the information needed to answer the question.

ChatGPT has drawn considerable attention since its prototype was released on November 30, 2022, with users sharing their impressions of the chatbot all across the globe. These researchers describe how the artificial intelligence (AI)-powered chatbot ChatGPT, as the first in a new line of language models, can prove to be an interactive medical education tool given its ability to represent a combination of clinical knowledge and dialogic interaction.

Conrad W Safranek, one of the medical students involved in the project, describes his use of ChatGPT as a study aid. Upon reflection, he found value in using the tool to unearth context relevant to the question, which supported his ability to recall external information and make logical connections from medical courses as expected by the question. Using this tool to enhance self-directed reflective learning is but one example of the opportunities that ChatGPT brings to enhancing medical education.

The corresponding author, David Chartash, PhD, from Yale University School of Medicine remarked, "JMIR Medical Education has proved time and again to understand the value of the integration of medical informatics in medical education. This study builds upon on the fundamental principles which I have previously written about with colleagues last year [see published work here]: as medical education seeks to develop competencies in clinical informatics for medical students, exposure to the fundamentals of novel technology in pre-clinical years that may shape their practice (such as with dialogic AI) will support their ability to understand the technology-augmented clinical practice they will inherit when they graduate."

The authors of the study believe that their results make a compelling case for the potential use of ChatGPT as an interactive medical education tool, as it provides users with contextually interpretable and narratively coherent translation of medical knowledge along with its answers. This study published by JMIR Publications marks a significant advancement in natural language processing models for medical question answering and could have a profound impact on the future learning environment for medical students.

To further demonstrate the capabilities of this tool, the authors asked ChatGPT to summarize their research findings. Want to know how ChatGPT performed? Read the (second) "Conclusions" of this paper here.

JMIR Medical Education Launches a Theme Issue on ChatGPT, Generative Language Models, and AI in Medical Education

Given the interest this research has generated among medical educators and researchers, JMIR Medical Education has released a call for papers for its upcoming theme issue and e-collection titled "ChatGPT and Generative Language Models in Medical Education" [2]. This special issue aims to explore the potential of emerging technologies like ChatGPT and similar generative language models or AI applications in medical education, including their use in teaching and learning, clinical decision-making, and patient care. JMIR Medical Education welcomes submissions from researchers, educators, and practitioners in medicine, health care, computer science, and related fields. Submissions from a breadth of professionals at all career stages who are engaged in medical education are welcome. We encourage both empirical and theoretical submissions, including original research, systematic reviews, viewpoints, and tutorials. We also encourage submissions that address practical challenges and opportunities related to the use of generative language models and AI in medical education.

"ChatGPT has changed the world and the potential for ChatGPT to disrupt medical education is significant," says Gunther Eysenbach, publisher at JMIR Publications. "ChatGPT not only provides new interactive learning opportunities for medical students and health professionals but also raises new interesting questions for medical educators. The special issue will be a useful resource for researchers, medical educators, and trainees alike to get the most out of this fascinating technology that will change how we teach and learn." In an accompanying editorial, Eysenbach interviews ChatGPT itself, having the machine illustrate some of the opportunities for medical education; however, some striking errors and limitations also became evident [3].

Cite as:

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment
JMIR Med Educ 2023;9:e45312 (Feb 8)
doi: https://doi.org/10.2196/45312 PMID: 36753318
Call for Papers for the ChatGPT Theme Issue for JMIR Medical Education: https://mededu.jmir.org/announcements/365
Eysenbach G. The Role of ChatGPT, Generative Language Models and Artificial Intelligence in Medical Education: A Conversation with ChatGPT and a Call for Papers
JMIR Med Educ 2023;9:e46885
http://preprints.jmir.org/preprint/46885
doi: https://doi.org/10.2196/46885

Contact:

Corresponding author: David Chartash, PhD

Institution: Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, US and School of Medicine, University College Dublin, National University of Ireland, Dublin, Dublin, IE

Email: [email protected]

About JMIR Publications

JMIR Publications is a leading, born-digital, open access publisher of 30+ academic journals and other innovative scientific communication products that focus on the intersection of health and technology. Its flagship journal, the Journal of Medical Internet Research, is the leading digital health journal globally in content breadth and visibility, and it is the largest journal in the medical informatics field.

To learn more about JMIR Publications, please visit https://www.JMIRPublications.com or connect with us via:

YouTube - https://www.youtube.com/c/JMIRPublications

Facebook - https://www.facebook.com/JMedInternetRes

Twitter - https://twitter.com/jmirpub

LinkedIn - https://www.linkedin.com/company/jmir-publications

Instagram - https://www.instagram.com/jmirpub/

Head Office - 130 Queens Quay East, Unit 1100 Toronto, ON, M5A 0P6 Canada

Media Contact - [email protected]

The content of this communication is licensed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, published by JMIR Publications, is properly cited.

Media Contacts: Ryan James Jessup JD/MPA
Publication Communications Specialist
Communications & Knowledge Translation Department
[email protected]
Mobile: (202) 638 9720

This news release was issued on behalf of Newswise^™. For more information, visit http://www.newswise.com

SOURCE JMIR Publications