Ember-2 (2.96M Parameters)
Ember-2 is a state-of-the-art, ultra-compact language model developed by SurjoLabs. It ranks #1 on the Open SLM Leaderboard for sub-3M parameter models with an Intelligence Index of 7.21, proving that recurrent weight-sharing and value-subtraction attention (XSA) can achieve exceptional general and mathematical reasoning at micro scales.
What's New in Ember-2?
- Increased Layer Count: Scaled from 8 physical layers (11 effective depth) to 10 physical layers (14 effective depth) by expanding to 4 recurrent layers and 5 coda layers, squeezing maximum expressive power into 2.96M parameters.
- Critical Dataset Fix: In the original Ember-1 pipeline, a configuration bug caused the FinePhrase dataset to map to raw source web data rather than the intended synthetically rephrased data. In Ember-2, this was fixed to properly train on the high-quality synthetic educational corpus.
- Length Filtering: Added strict quality filtering to FinePhrase (length > 24 tokens) to eliminate noisy short fragments and low-information sequences.
- Performance Leap: Jumped from 5.94 -> 7.21 on the Open SLM Intelligence Index, notably achieving 35.90% on ArithMark-3 and 55.11% on PIQA.
Methodology
Ember-2 is built to demonstrate that specialized recurrent architectures can dramatically outperform standard transformers under extreme parameter constraints (under 3M parameters).
- Custom Architecture: Built on a Llama-variant framework featuring XSA Attention (value-subtraction projection) and Recurrent Weight-Tied Layers. By passing hidden states twice through its 4 recurrent layers, the model achieves the effective computational depth of a 14-layer model (1 prelude + 4x2 recurrent + 5 coda) while only storing 10 physical layers of weights (~2.96M total parameters).
- Optimized Tokenizer: Uses a custom 4,096-vocabulary English tokenizer. Keeping the vocabulary small prevents embedding tables from dominating the budget, ensuring ~80% of parameters are dedicated strictly to attention and MLP compute.
- Extreme Overtraining: Trained across 20 Billion tokens (~1:6,750 parameter-to-token ratio), driving dense knowledge compression and syntax mastery far past standard compute-optimal thresholds.
- Data Mixture:
- 60% FinePhrase (Synthetic FineWeb-Edu, filtered with length > 24 tokens)
- 20% DCLM (Curated web text)
- 10% FineMath (Mathematical reasoning)
- 10% CornStack Python (Python code)
- Optimizer: Hybrid setup pairing Muon (for 2D weight matrices) with AdamW (for embeddings and normalization layers) to guarantee stable convergence during high-token training.
Benchmark Results (Open SLM Leaderboard)
Evaluated at 0-shot using normalized accuracy (acc_norm):
| Benchmark | Ember (v1) | Ember-2 |
|---|---|---|
| HellaSwag | 27.35% | 27.28% |
| ARC-Easy | 33.25% | 33.42% |
| ARC-Challenge | 21.08% | 22.01% |
| PIQA | 54.41% | 55.11% |
| ArithMark-3 | 32.90% | 35.90% |
| Intelligence Index | 5.94 | 7.21 (#1) |
- Evaluations for HellaSwag, ARC-Easy, ARC-Challenge, and PIQA were performed using lm-evaluation-harness (0-shot, acc_norm).
- ArithMark-3 evaluated using the official benchmark evaluation script (acc_norm).
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SurjoLabs/Ember-2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgements
We would like to thank AxiomicLabs for pioneer work proving the token-efficiency of the XSA attention mechanism.
Limitations
Ember-2 is an experimental sub-3M parameter research model developed under the broader Surjo Project. While it demonstrates strong reasoning for its size, output fluency is naturally constrained by parameter capacity. The custom modeling implementation is intended for research and benchmark evaluation.