TinyLlama

After much anticipation, the TinyLlama project has released a remarkable open source model. The project began last September with developers working to train a small model on trillions of tokens. After some hard work and some setbacks, the TinyLlama team has now released the model. The model has 1 billion parameters and runs about three epochs, or three cycles through the training data. The final version of TinyLlama outperforms existing open source language models of similar size, including Pythia-1.4B…