OLMo

AI2's newly released Open Language Model (OLMo) framework aims to promote research and experimentation with large-scale language models. By providing training code, models, and evaluation code on Hugging Face and GitHub, AI2 is committed to enabling academia and researchers to jointly study the science of language models, explore the impact of new pre-training data subsets on downstream performance, and study new pre-training methods and stability. The first batch of models in the project include four 7B scale final variants corresponding to different architectures, optimizers, and training hardware, and a 1B scale model, all of which are...