min-spark 1.5
min-spark 1.5 is a 9.36M-parameter language model with native effort levels. It reaches an Intelligence Index of 9.79, higher than any sub-10M model currently on the Open SLM leaderboard, after training on just 2.0B tokens.
What's new in 1.5
Much better results for far less training. min-spark 1.5 beats min-spark 1.1 while training on 8× fewer tokens with 5.7× less compute. It also beats Hummingbird-V2, the strongest other sub-10M model on the leaderboard, which trained on 10B tokens, five times as many.
| min-spark 1.1 | min-spark 1.5 | |
|---|---|---|
| Int Index | 9.44 | 9.79 |
| Training tokens | 16.81B | 2.0B (8× fewer) |
| Training compute | 2.2 × 10¹⁸ FLOPs | 3.9 × 10¹⁷ FLOPs (5.7× less) |
| Parameters | 5.76M | 9.36M |
min-spark runs a few shared transformer blocks several times in a row. Each pass has its own small adapter, so it can do something different from the pass before. In earlier versions those adapters were held back. Version 1.5 fixes that:
- Adapters on every layer. Every pass now adapts all four projections of each shared block instead of only the attention input, at rank 24 instead of 16.
- Stable adapters. Adapters read normalized inputs and train at a steadier learning rate, so none of them can grow out of control and shut off part of a block.
- No wasted passes. Earlier versions trained a fourth pass that the model learned to ignore. Version 1.5 trains three, and all three do real work.
- A bigger core and new data. There are five shared blocks instead of three. Cosmopedia v2 textbooks and a STEM web crawl make up a fifth of the training data.
Generation also gets faster: 1.5 adds a KV cache, which makes generation about 3× faster on CPU.
Effort levels
Effort levels set how many passes the model makes through its shared core. Large reasoning models usually do this by changing a thinking-token budget. min-spark changes the computation inside the network instead.
| Effort | Passes | Use |
|---|---|---|
low | 2 | Fastest completion |
medium | 3 (default) | General use; best on most tasks |
high | 4 | Experimental: one pass more than the model was trained with. It still gives the best HellaSwag score |
| Effort | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | ArithMark-3 | Int Index |
|---|---|---|---|---|---|---|
| min-spark 1.5 low | 37.21% | 23.55% | 28.11% | 56.69% | 34.30% | 8.98 |
| min-spark 1.5 medium | 38.64% | 22.70% | 28.15% | 57.34% | 34.90% | 9.59 |
| min-spark 1.5 high | 36.99% | 23.21% | 28.27% | 56.58% | 34.00% | 8.80 |
| best per task | 38.64% | 23.55% | 28.27% | 57.34% | 34.90% | 9.79 |
Evaluation
All scores are zero-shot and length-normalized. ARC, HellaSwag and PIQA use lm-eval 0.4.12. ArithMark-3 uses the official ArithMark-3.0 script. As on earlier min-spark cards, each benchmark is reported at its best effort level.
The Intelligence Index is the Open SLM leaderboard's summary score. Each benchmark is rescaled so that random guessing scores 0 and a perfect score is 100, then averaged, with ARC as the mean of Easy and Challenge:
Int Index = (HellaSwag + ARC + PIQA + 0.65 × ArithMark-3) / 3.65
The table below compares 1.5 with the leading sub-10M models on the leaderboard that have a public model card. Peer scores come from the leaderboard and training tokens from each model's card.
| Model | Params | Training tokens | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | ArithMark-3 | Int Index |
|---|---|---|---|---|---|---|---|---|
| min-spark 1.5 | 9.36M | 2.0B | 38.64% | 23.55% | 28.27% | 57.34% | 34.90% | 9.79 |
| Hummingbird-V2 | 9.59M | 10B | 39.65% | 21.16% | 27.61% | 57.40% | 36.00% | 9.59 |
| min-spark 1.1 | 5.76M | 16.81B | 36.45% | 23.04% | 28.24% | 57.40% | 35.40% | 9.44 |
| ForgePlex-M2-9M | 9.95M | 30B | 36.53% | 23.46% | 28.02% | 57.18% | 34.60% | 9.14 |
| Spark-2A | 9.15M | 34.26% | 22.18% | 28.13% | 57.24% | 37.00% | 9.14 |
Usage
min-spark 1.5 loads through Transformers with remote code:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"MinimaLabs/min-spark-1.5",
trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
"MinimaLabs/min-spark-1.5",
trust_remote_code=True,
)
inputs = tokenizer("The meaning of life is", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, effort="medium", max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
effort works the same way with the pipeline API. To skip Transformers, use the bundled script:
python generate.py -p "The meaning of life is" -e medium
Generation runs one sequence at a time and uses the KV cache by default. Right-padded batches work for scoring.
Architecture
| Field | Value |
|---|---|
| Parameters | 9,360,596 |
| Architecture | Meiosis (tied-embedding looped decoder) |
| Vocabulary | 4,096-token byte-level BPE |
| Embedding width | 288 |
| Heads | 6 query · 2 KV (GQA) |
| FFN hidden size | 832 |
| Blocks | 1 prelude · 5 shared body blocks · 1 coda · final RMSNorm |
| Per-pass adapters | LoRA rank 24 on attention QKV, attention output, FFN gate/up and FFN down |
| Context window | 512 tokens |
| Effort (loop count) | low = 2 · medium = 3 · high = 4 (trained up to 3) |
Training
| Field | Value |
|---|---|
| Training tokens | 2.0B, from scratch |
| Precision | bf16 autocast |
| Global batch size | 128 sequences × 512 tokens |
| Optimizer | Muon (matrices) + NAdamW (other parameters), weight decay 0.01 |
| Learning-rate schedule | 2,000-step warmup, stable phase, linear decay to 10% over the final 20% |
| Objective | Next-token loss at 3 passes, plus 0.1 × the loss at 2 passes and 0.1 × a term that keeps the 2-pass state close to the 3-pass one |
| Attention masking | Intra-document |
| Data source | Share |
|---|---|
| FineWeb-Edu (score 4+) | ~63% |
| Cosmopedia v2 (synthetic textbooks), new in 1.5 | 15% |
| DCLM-baseline filtered for commonsense, plus WikiHow | ~11% |
| FineMath (finemath-4plus) | ~5% |
| Dolmino STEM crawl, new in 1.5 | 4% |
| Synthetic bracket-matching sequences, at the start of training | ~1% |
Both new sources were decontaminated: any document sharing a 13-word sequence with an item from the five evaluation sets was dropped.
Reproducing the evaluation
python run_lmeval.py --effort low,medium,high
python bencharithmark-3.py --model MinimaLabs/min-spark-1.5 --dtype float32
The first command scores ARC, HellaSwag and PIQA at each effort and prints each task's best (add --limit N for a quick run). The second is the official ArithMark-3 script, which runs at the default medium effort.
Limitations
This is a small base model. It has not been instruction-tuned, so it has no conversational behavior, alignment or safety filtering, and its knowledge and reasoning are limited by its size. The context window is 512 tokens; all results were measured within it.
License
Apache-2.0. See LICENSE.