ChatGPT, an artificial intelligence language bot that can generate text in response to user-inputted prompts, has passed parts of high-level post-graduate coursework in business, law and medical programs.
The AI “chatbot” was able to achieve passing marks in parts of the U.S. Medical Licensing Exam (USMLE), according to a report from medical news site MedScape.
ChatGPT also passed law exams in four courses at the University of Minnesota and another exam at University of Pennsylvania’s Wharton School of Business, according to professors at the school who spoke with CNN.
For the USMLE exam, doctors at Ansible Health prompted ChatGPT on hundreds of multiple-choice questions covered by the test.
The doctors first gave the chatbot a theoretical patient’s signs and symptoms and asked it to determine what may be the underlying cause, according to MedScape.
They then refreshed ChatGPT to remove any potential bias from retained information and asked it to answer multiple-choice questions. After answering those questions it was refreshed again, and the chatbot was asked to clarify why the correct answers were correct and the incorrect answers were not.
Three board-certified, licensed physicians then reviewed how the AI performed on the exam’s Step 1, Step 2CK (clinical knowledge) and Step 3.
“ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement,” researchers wrote in the pre-print of their research article in December. “Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.”
The article, titled “Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models,” has since been accepted for peer review at the journal PLOS Digital Health.
A doctor who helped author the article told MedScape it will be published soon.
For the law courses at the University of Minnesota, ChatGPT performed at the level of a C+ student after completing 95 multiple-choice questions and 12 essay questions.
While the researchers said the results were not exemplary, the chatbot did pass all four courses.
ChatGPT earned a B during its exam for Wharton’s business management course. The AI did particularly well at answering basic operations and process-analysis questions; however, it struggled with advance prompts and made mistakes with some of the basic math.