XVerse

XVERSE released XVERSE-MoE-A36B, the largest MoE open source model in China. The model has 255B total parameters and 36B activation parameters, and it is officially claimed that the effect can "roughly reach" more than the "cross-level" performance leap of the 100B large model, while the training time is reduced by 30%, and the inference performance is improved by 100%. "At the same time, the training time is reduced by 30%, the inference performance is improved by 100%, and the cost per token is greatly reduced. MoE (Mixture of Experts) is a hybrid expert modeling architecture that combines multiple segmented expert models into a single super model, expanding the model size...