YaFSDP

Russian technology giant Yandex has launched an open-source large language model training tool, YaFSDP, which claims to be up to 26% faster than existing tools. According to reports, YaFSDP outperforms the traditional FSDP method in terms of training speed, especially for large models. In terms of pre-training LLM, YaFSDP is 20% faster and performs better under high memory pressure conditions. For example, YaFSDP can achieve 21% efficiency improvement for Llama 2 with 70 billion parameters, and 100% faster for Llama 2 with 70 billion parameters.