Rigel is a 2.3B-total, 360M-active-parameter hybrid Mamba-2 + attention mixture-of-experts base model, trained with LM Engine. It used about 117× fewer pretraining FLOPs than Llama-3.2-1B/3B and 53× fewer than Granite-4.2-3B, while landing within a few points of Granite-4.2-3B and Llama-3.2-3B on zero-shot averages and above Llama-3.2-1B.
Read the full write-up, including architecture, data curriculum, the multi-chip training infrastructure, and evaluations: Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute.
This repo is the 294,912-token long-context extension, see open-lm-engine/rigel-base-long.
Model summary
| Total / active parameters | 2.3B / 360M (260M active non-embedding) |
| Layers | 40 (30 Mamba-2, 10 grouped-query attention) |
| Attention | GQA (16 heads, 4 KV heads), 4,096-token sliding window, exclusive self attention (XSA), no positional embeddings |
| Feed-forward | Fine-grained MoE: 128 SwiGLU experts, intermediate size 128, top-2 routing |
| Hidden size | 1,024 |
| Vocabulary | 100,352 tokens, tied input/output embeddings |
| Pretraining context | 4,096 tokens |
This is a base model: it has no chat template and is not instruction-tuned.
Usage with LM Engine
Rigel's architecture lives in LM Engine. Installing it and importing lm_engine.training registers the architecture with transformers' Auto classes, after which the usual Hugging Face API works.
git clone https://github.com/open-lm-engine/lm-engine.git && cd lm-engine && uv sync --extra cuda
import torch
import lm_engine.training # registers Rigel's architecture with transformers' Auto classes
from lm_engine.training.kernels import Kernel, enable_kernels
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "open-lm-engine/rigel-mid"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map={"": "cuda"}).eval()
inputs = tokenizer("Rigel is a blue supergiant star in the constellation", return_tensors="pt").to("cuda")
with torch.no_grad(), enable_kernels([Kernel.mamba2_ssm, Kernel.causal_conv1d, Kernel.sonicmoe, Kernel.flash_attention_3]):
output_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Any kernel left out of the list falls back to a pure-PyTorch implementation, so the snippet also runs with an empty list, just slower.
To finetune or continue pretraining, point an LM Engine training config at this checkpoint; see the LM Engine README.
Citation
@misc{mishra2026rigel,
title = {Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute},
author = {Mishra, Mayank and Runwal, Bharat and Stoica, Ion and Dao, Tri and Gonzalez, Joseph E.},
year = {2026},
url = {https://open-lm-engine.github.io/blog/rigel/}
}