According to the official WeChat account of Shanghai Sixth People's Hospital, the team of Professor Jia Weiping and Professor Li Huating from the Shanghai Sixth People's Hospital affiliated to Shanghai Jiao Tong University School of Medicine, the team of Professor Sheng Bin from the Department of Computer Science, School of Electrical Engineering, Shanghai Jiao Tong University/Key Laboratory of Artificial Intelligence, Ministry of Education, together with the team of Professor Huang Tianyin from Tsinghua University and the team of Professor Qin Yuzong from the National University of Singapore, have established a cross-disciplinary medical and engineering research project.The world's first multimodal integrated intelligent system of vision-large language model for diabetes diagnosis and treatment DeepDR-LLMThe results were published in Nature Medicine on July 19, 2024 (titled Integrated image-based deep learning and language models for primary diabetes care).
According to reports, the DeepDR-LLM system combines the advantages of large language models and deep learning technology to achieveMultimodal generation of medical imaging diagnosis and treatment opinionsFunction, can provideDiabetic retinopathy auxiliary diagnosis resultsandPersonalized comprehensive diabetes management advice.
The system was retrospectively validated in a multicenter cohort covering seven countries in Asia, Africa and Europe, and prospectively validated in real-world settings for China's primary care. This is the first time that the system has provided the world with a comprehensive and reliable data set for the diabetes healthcare vertical.Multimodal large modelApplication effectivenessHigh-quality evidenceThe research team said that the study proved that the DeepDR-LLM system canEffectively improve DR screening and primary diabetes management, providing a basis for future global diabetes governanceRevolutionary digital solutions.
Previous AI system research and development has mainly focused onDiabetes complications screening or auxiliary managementWith the rapid development of generative artificial intelligence technology represented by ChatGPT around the world, multimodal large models are constantly promoting the emergence of new application scenarios and models in the medical field, but these modelsNot yetProvide accurate and safe diabetes care based on the patient's medical images and specific conditionsComprehensive diagnosis and treatment recommendations.
In response to the current technical gaps and actual clinical needs, the team successfully developed the world's first visual-large language model multimodal integrated intelligent system DeepDR-LLM for primary diagnosis and treatment of diabetes. It can be adapted to include LLaMA The LLM module integrates the training network layer with the inherent weight parameters of the large language model.Breaking through the bottleneck of multi-modal large model optimization under low computing resource constraints, and further based on 372,000 pieces of data and knowledge on diagnosis and treatment of chronic diseases and management of chronic diseases at the grassroots levelEfficient optimization training has been achieved, enabling the DeepDR-LLM system to generate accurate diabetes management advice based on individual patients’ clinical information.
Attached research results:
https://www.nature.com/articles/s41591-024-03139-8