ChatGPT-diagnosed patients were at least as likely to rush to the emergency room as doctors. (Representative)
The Hague the Netherlands:
Artificial intelligence chatbot ChatGPT diagnosed patients rushed to emergency rooms at least as well as doctors and in some cases outperformed them, Dutch researchers have found.
But the report published on Wednesday also highlighted that ER doctors do not need to hang up their surgeries yet, with the chatbot potentially able to speed up diagnosis but not replacing human medical judgment and experience.
Scientists examined 30 cases treated at an emergency room in the Netherlands in 2022, adding anonymized patient history, laboratory tests and the doctors’ own observations to ChatGPT, asking for five possible diagnoses.
They then compared the chatbot’s shortlist with the same five diagnoses suggested by ER doctors with access to the same information, then checked the correct diagnosis in each case.
Doctors got the correct diagnosis in the top five 87 percent of the time, compared to 97 percent for ChatGPT version 3.5 and 87 percent for version 4.0.
“Simply put, this indicates that ChatGPT was able to make medical diagnoses just as a human doctor would,” said Hidde ten Berg of the emergency medicine department at the Dutch Jeroen Bosch Hospital.
Co-author Steef Kurstjens told AFP that the study did not indicate that computers could ever run the ER, but that AI could play a crucial role in assisting doctors under pressure.
“The most important point is that the chatbot does not replace the doctor, but it can help make a diagnosis and perhaps come up with ideas that the doctor has not yet thought of,” Kurstjens told AFP.
Large language models like ChatGPT are not designed as medical devices, he pointed out, and there would also be privacy concerns about entering confidential and sensitive medical data into a chatbot.
‘Bloopers’
And like other areas, ChatGPT showed some limitations.
The chatbot’s reasoning was “sometimes medically implausible or inconsistent, which can lead to misinformation or incorrect diagnoses, with significant consequences,” the report said.
The scientists also admitted some shortcomings in the study. The sample size was small: 30 cases were examined. In addition, only relatively simple cases were examined, in which patients had only one primary complaint.
It was not clear how well the chatbot would do with more complex matters. “ChatGPT’s efficacy in providing multiple discrete diagnoses for patients with complex or rare diseases remains unverified.”
Sometimes the chatbot did not provide the correct diagnosis in the top five possibilities, Kurstjens explains, especially in the case of an abdominal aneurysm, a potentially life-threatening complication in which the aortic artery swells.
The only consolation for ChatGPT: in that case too, the doctor was wrong.
The report details what it calls the medical “bloopers” the chatbot has created, for example diagnosing anemia (low hemoglobin levels in the blood) in a patient with normal hemoglobin levels.
“It is vital to remember that ChatGPT is not a medical device and there are privacy concerns when using ChatGPT with medical data,” ten Berg concludes.
“However, there is potential here to save time and reduce waiting times in the emergency department. The benefit of using artificial intelligence could be to support doctors with less experience, or it could help detect rare diseases,” he added.
The findings – published in the medical journal Annals of Emergency Medicine – will be presented at the European Emergency Medicine Congress (EUSEM) 2023 in Barcelona.
(Except for the headline, this story has not been edited by DailyExpertNews staff and is published from a syndicated feed.)
