January 14th.New York UniversityA study reveals that large-scale language models (LLM) potential risks in medical information training. Studies have shown thateven ifTraining Datacontaining only 0.001% of error information may also cause the model to output inaccurate medical answers.
Data "poisoning" is a relatively simple concept; LLMs are usually trained on large amounts of text, mostly from the Internet. By injecting specific information into the training data, it is possible for the model to treat this information as fact when generating answers. This approach does not even require direct access to the LLM itself.Simply posting the target information on the Internet may be included in the training data. For example, a pharmaceutical company may be able to influence a model's perception of a particular drug simply by releasing a few targeted documents.
According to 1AI, the research team chose "The Pile", a database commonly used for LLM training, as the object of study. The database contains a large amount of medical information, of which about a quarter of the sources are not human-vetted and are mainly crawled from the Internet. The researchers selected 60 topics in three medical domains (general medicine, neurosurgery, and pharmaceuticals) and embedded "high quality" medical misinformation generated by GPT-3.5 in "The Pile". The results show that even when replacing only 0.5% to 1% of relevant information, the probability of the trained model generating misinformation on these topics increases significantly, and the misinformation affects other medical topics as well.
The researchers further explored the minimum impact threshold for misinformation. In the case of vaccine misinformation, for example, even if the misinformation accounted for only 0.011 TP3T of the training data, more than 101 TP3T of the answers generated by the model contained misinformation;When the percentage of incorrect information drops to 0.001%, there are still more than 7% of answers that are harmfulThe researchers noted that a similar attack against the LLaMA 2 model, which has 70 billion parameters, would be possible with only 40,000 articles (at a cost of less than $100). The researchers note that a similar attack against the LLaMA 2 model, which has 70 billion parameters, could generate just 40,000 articles (at a cost of less than $100). These "articles" could be ordinary web pages, with error messages placed in areas of the page that would not be normally viewed, or even by hiding text (e.g., black text on a black background).
The study also points out that the problem of existing misinformation cannot be ignored as well. Many laypeople tend to get their medical information from generic LLMs, and these models are often trained based on the entire Internet, which contains a lot of unvetted misinformation. The researchers devised an algorithm that recognizes medical terms in the output of LLMs and cross-references them with validated biomedical knowledge graphs to flag unverifiable phrases. While this approach failed to capture all medical misinformation, it successfully flagged much of it.
However, even the best medical databases (e.g., PubMed) suffer from the problem of misinformation. The medical research literature is filled with unrealized theories and obsolete treatments.
Studies have shown that even when relying on the highest quality medical databases, there is no guarantee that a trained LLM will be completely immune to misinformation. The complexity of the medical field makes it especially difficult to create a consistently reliable medical LLM.
