Maba-v1-600M is a 752M-parameter causal language model (498M non-embedding) trained on ~150 billion tokens. It uses a 3:1 hybrid design: 75% Gated DeltaNet-2 linear recurrence layers and 25% Grouped-Query Attention layers. This structure provides O(N) context scaling and cuts KV-cache memory requirements by 75% compared to standard Transformers.
Hardware implementations, CUDA kernels, and theoretical derivations are documented in the Maba Architecture Repository.
Benchmarks
Evaluations were conducted with lm-evaluation-harness in float16 precision with batch size 4. Total runtime was 2 hours 1 minute across 104,876 test items.
Note on dual-pass inference: Dual-pass execution was disabled in this release due to a parameter bug in the training script. This was caught when training had already practically finished. All benchmark numbers reported above reflect standard single-pass inference. Dual-pass evaluation will be released in an upcoming checkpoint.
Benchmark Comparison
| Benchmark | Maba-v1-600M | Qwen3-0.6B | SmolLM2-360M | Llama-3.2-1B |
|---|---|---|---|---|
| MMLU (Academic Knowledge, 0-shot) | 49.2% | 47.2% | 35.8% | 49.3% |
| GSM8K (Math Reasoning, 5-shot) | 32.5% | 43.0% | 3.2% | 26.2% |
| ARC-Challenge (Science Reasoning, 0-shot) | 38.1% | 42.3% | 35.8% | 46.2% |
| Winogrande (Context & Logic, 0-shot) | 58.6% | 59.2% | 52.5% | 61.2% |
| HellaSwag (Common Sense, 0-shot) | 50.5% | 53.8% | 54.5% | 68.8% |
All Maba evaluations were executed via lm-evaluation-harness in float16 precision.
MMLU Performance by Domain
| Domain | Accuracy |
|---|---|
| Social Sciences | 55.31% |
| Applied & Professional | 53.04% |
| STEM | 45.54% |
| Humanities | 45.14% |
Top MMLU Disciplines
| Discipline | Accuracy |
|---|---|
| Marketing | 77.78% |
| International Law | 74.38% |
| US Foreign Policy | 69.00% |
| Management | 67.96% |
| High School Psychology | 67.89% |
Specifications
| Attribute | Value |
|---|---|
| Total Parameters | 752M |
| Non-Embedding Parameters | 498M |
| Layers | 24 (18 Gated DeltaNet-2 + 6 GQA) |
| Hidden Size | 1024 |
| Intermediate Size (SwiGLU) | 3584 |
| Attention Heads | 16 Query / 2 Key-Value (GQA) |
| Vocabulary Size | 248,320 |
| Context Length | Up to 32,768 tokens |
| Pretraining Volume | ~150B tokens |
Quickstart
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AndrewThompson1233/maba-v1-600m"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True
)
messages = [
{"role": "system", "content": "You are Maba, an assistant built on the Maba v1 architecture."},
{"role": "user", "content": "Explain the difference between linear attention and quadratic attention."}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7,
top_p=0.9
)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
Architecture Reference
For architecture design, training dynamics, benchmarks against quadratic baselines, and implementation details, visit the Maba Architecture Repository.