hybrid-moe-30b-a3b-base
A base language model with a hybrid architecture of Mamba2, attention and mixture-of-experts layers, trained from random initialisation.
| Architecture | Hybrid Mamba2 + attention + mixture-of-experts (model_type: nemotron_h) |
| Layers | 52 |
| Hidden size | 2,688 |
| Parameters | 31.6B total, roughly 3B active per token |
| Initialisation | Random (trained from scratch) |
| Tokens seen | 553,765,568,512 (553.8B) |
| Context length | Trained at 8,192 tokens, then continued at 32,768 |
| Precision | BF16 |
This is a base model: it has had no instruction tuning or preference optimisation, so it is suited to text completion rather than chat.