Nowadays, small language models are becoming popular, and many manufacturers have begun to launch "Small Model”, this week Hugging Face It announced the "SmolLM"A family of small language models, including 135 million, 360 million, and 1.7 billion parameter models.
According to reports, these models are said to be trained with carefully planned high-quality training data sets, and are said to be quite powerful in Python programming performance. The team pointed out that they focused on optimizing the amount of RAM required for the model, "even on an iPhone 15 with 6GB of RAM."
In terms of training, the Hugging Face team first created a dataset called SmolLM-Corpus (click here to access the dataset address), which mainly includes Python teaching content Python-Edu, Web education content FineWeb-Edu, and common sense content generated by the Mixtral-8x7B-Instruct-v0.1 and Cosmopedia v2 models, with a total token volume of 600 billion. After that, the Hugging Face team used the SmolLM-Corpus dataset to train the "SmolLM" small language model.
The Hugging Face team benchmarked the SmolLM model they developed against other models with the same number of parameters. The SmolLM-135M surpassed other models with less than 200 million parameters in multiple tests. The SmolLM-360M performed better than all models with less than 500 million parameters, but was inferior to the MobileLLM-350M just announced by Meta in some projects. The SmolLM-1.7B model surpassed all models with less than 2 billion parameters, including Microsoft Phi-1.5, MobileLLM-1.5B and Qwen2.
