In a new study, scientists at Beth Israel Deaconess Medical Center (BIDMC) compared a large language model with human doctors to find out whether the patient's speech was correct or not.Clinical ReasoningComparison of abilities. The researchers used the revised IDEA (r-IDEA) score, a commonly used tool for assessing clinical reasoning abilities.
This study involved giving GPT-4The chatbot, 21 attending physicians, and 18 residents were asked to provide support for 20 clinical cases to build diagnostic reasoning and solve problems. The r-IDEA scores of the three groups of answers were then evaluated. The researchers found that the chatbot actually gainedHighestThe authors found that the chatbot “often got it completely wrong”.
Source Note: The image is generated by AI, and the image is authorized by Midjourney
"Further research is needed to determine how large language models can be used to predict the future of speech," explains lead author Dr. Stephanie Cabral.most“In summary, the results showed reasonable reasoning by the chatbot, but also significant errors; this further supports that this AI-driven system, at its current level of maturity, is best suited as a tool to augment physicians’ practice rather than replace their diagnostic abilities.”
As medicalLeadersAs technologists often explain, this is because the practice of medicine is not based solely on the output of rule-based algorithms, but on deep reasoning and clinical intuition, which is LLM However, tools like this that provide diagnostic or clinical support can still be extremely powerful assets in the physician’s workflow. For example, if the system can reasonably provide “firstDiagnosis” or preliminary diagnosis suggestions may allow doctors to save a lot of time in the diagnosis process. In addition, there may be opportunities to increase efficiency if these tools can enhance doctors’ workflow and improve their ability to process the large amounts of clinical information in medical records.
Many organizations are taking advantage of these potential clinical enhancements. For example, AI-driven transcription technologies that leverage natural language processing are helping physicians complete clinical documentation more efficiently. Enterprise search tools are integrating with organizational and electronic medical record systems to help physicians search large amounts of data, promote data interoperability, and gain faster and deeper insights into existing patient data. Other systems may even help provide preliminary diagnoses; for example, tools are emerging in the fields of radiology and dermatology that can suggest potential diagnoses by analyzing uploaded photos.
However, there is still much work to be done in this area. In short, although these AI systems are not yet ready for clinical diagnosis, it is still possible to use this technology to enhance clinical workflow, especially to ensure safe and accurate processes while maintaining human control.