Generative Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including question answering and dialogue systems. However, most models are trained on English data and lack strong generalization capabilities in providing Chinese answers.
To address this problem, the research team introducedMedChatZH, a conversational model optimized for TCM question answering, based on the transformer decoder of the LLaMA architecture. Fine-tuned on a carefully selected medical guidance dataset, MedChatZH significantly outperforms several Chinese conversational baseline models on a real-world medical conversational dataset. Our model, code, and dataset are all publicly available on GitHub to encourage further research and development in the field of traditional Chinese medicine and LLMs.
Currently, large language models perform well in various downstream natural language processing tasks, including machine translation and entity recognition. In 2023, OpenAI launched the ChatGPT series of generative dialogue models, which caused a major response in academia and industry, and significantly promoted the development of the field of natural language technology. However, most language models, including ChatGPT, are mainly trained and fine-tuned on English datasets. When applied to other languages (such as Chinese), they often encounter limitations due to vocabulary mismatch, differences in corpus context, and answer habits, resulting in reduced response quality and generalization ability. In addition, the problem of lack of domain-specific understanding in general language models is particularly prominent in professional fields, such as traditionalChinese medicine consultationEven models like ChatGPT have difficulty with this task due to the following reasons: strict ethical and review constraints in the medical field, which lead to the refusal to provide medical advice; and the lack of TCM training data, which leads to insufficient understanding of TCM-related queries.
To address the domain adaptation challenges of LLMs in Chinese application scenarios, researchers have proposed a set of new base models, including Chinese-LLaMA and BLOOM, which are extended and trained on Chinese vocabulary, and bilingual models such as GLM. However, applying general language models directly to professional domains has not yet achieved the desired performance level. Compared with the rich and diverse English corpus, Chinese corpora are often isolated in various commercial software, not easily publicly accessible, or suffer from poor quality and lack of peer review on the Internet. This leads to the scarcity of Chinese pre-training corpora and downstream task data, which is the main obstacle to training general Chinese LLMs. The challenge of applying general language models to specific verticals in Chinese, such as TCM consultation dialogues, is even more prominent. Even models such as ChatGPT face difficulties in this task due to the following reasons: strict ethical and review constraints in the medical field, which often lead to the refusal to provide medical advice; the lack of TCM training data, which leads to insufficient understanding of TCM-related queries.
To address the research gap in the field of TCM QA and enhance conversational capabilities, this paper introduces a generative LLM, MedChatZH, optimized specifically for TCM patient conversations. It is first trained on a large amount of TCM literature, giving it a comprehensive and reliable foundation in traditional TCM knowledge. It is then fine-tuned on a high-quality dataset that combines medical and general conversations, and undergoes complex data processing steps such as heuristic and model-based filtering to remove irrelevant or sensitive content, such as personal information and spoken responses. The performance of MedChatZH is tested on a real-world TCM QA benchmark dataset, with evaluation metrics covering traditional and reward-based model scoring. The results show that MedChatZH outperforms TCM conversational LLMs such as BenTsao (formerly known as HuaTuo) and HuatuoGPT.
The main contributions of the paper are as follows:
1. We propose MedChatZH, a generative AI system designed specifically for traditional Chinese medicine consultation dialogues, which performs well.
2. We collected and organized a large amount of TCM literature, built a pre-training corpus, and constructed a high-quality dataset that combines general conversations and medical conversations. Through heuristic and reward evaluation, we filter out sensitive information and low-quality spoken responses.
3. The performance of MedChatZH is evaluated on a real-world TCM question-answering benchmark dataset, demonstrating its superiority over other baseline models in multiple evaluation metrics.
MedChatZH excels in managing the unique complexity of traditional Chinese medicine conversations, which can be attributed to its extensive pre-training on a large body of TCM literature and fine-tuning on a dataset combining medical and general conversations. The effectiveness of MedChatZH is expected to open new avenues for LLMs specific to different linguistic and cultural contexts.
Paper address: https://www.sciencedirect.com/science/article/abs/pii/S0010482524003743?dgcid=rss_sd_all