Jamba

AI21 released the world's first production-level model of Mamba: Jamba. This model uses the groundbreaking SSM-Transformer architecture with 52B parameters, of which 12B are active at generation time. Jamba combines Joint Attention and Mamba technology to support 256K context length. A single A10080GB can accommodate up to 140K context. Compared with Mixtral8x7B, the throughput of long context is increased by 3 times. Model address: https://huggingf…