In the field of artificial intelligence, developers and users have been challenged by the need for more customized and nuanced responses from large language models. While these models, such as Llama2, can generate human-like text, they are often required to provide answers that are truly tailored to the unique needs of individual users. However, existing methods, such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), have some limitations that result in generated responses that can appear mechanical and complex.
Source Note: The image is generated by AI, and the image is authorized by Midjourney
NVIDIAThe research department has introduced SteerLM, a breakthrough technology designed to address these challenges. SteerLM provides a novel user-centric approach for customizing the responses of large language models, allowing users to define key properties that guide the model's behavior. SteerLM operates through a four-step supervised fine-tuning process, simplifying the customization process of large language models.
First, it trains an attribute prediction model using a manually annotated dataset to assess characteristics such as helpfulness, humor, and creativity. Next, it leverages this model to annotate a variety of different datasets, enriching the diversity of data accessible to language models. SteerLM then employs attribute-conditioned supervised fine-tuning to train the model to generate responses based on specified attributes, such as perceived quality. Finally, it refines the model through guided training to generate a variety of different responses to achieveoptimalof fine-tuning.
A notable feature of SteerLM is its real-time adjustability, allowing users to fine-tune properties during inference to meet their specific needs. This flexibility opens the door to a variety of potential applications, from gaming and education to accessibility. With SteerLM, companies can provide personalized capabilities to multiple teams without having to rebuild the model for each different application.
SteerLM's simplicity and user-friendliness are reflected in its metrics and performance. In experiments, SteerLM43B outperformed existing RLHF models such as ChatGPT-3.5 and Llama30B RLHF on the Vicuna benchmark. By providing a simple fine-tuning process that requires almost no drastic changes to infrastructure and code, SteerLM can deliver excellent results with less hassle, making it a major advance in the field of AI customization.
NVIDIA is promoting advanced customization by releasing open source software for SteerLM in its NVIDIA NeMo framework. Developers now have the opportunity to access the code and try out this technology using a customized 13B Llama2 model available on platforms such as Hugging Face.
Official blog description: https://blogs.nvidia.com/blog/2023/10/11/customize-ai-models-steerlm/?ref=maginative.com