Ember (2.87M Parameters)
Ember is a highly efficient, ultra-small language model developed by SurjoLabs. It ranks #2 on the Open SLM Leaderboard for sub-3M parameter models, demonstrating that this custom XSA recurrent architecture is viable and can yield exceptional reasoning capabilities in tiny architectures.
Methodology
Ember is built to prove that unique recurrent architectures can match or beat standard transformers even at the smallest scales.
- Custom Architecture: Built on a Llama-variant framework but features XSA Attention (value-subtraction projection) and Recurrent Layers. By reusing weights across recurrent passes, the model achieves the depth of an 11-layer model while only carrying 8 layers of unique weights.
- Optimized Tokenizer: Uses a custom 4,096-vocabulary English tokenizer. This intentionally small vocabulary prevents the embedding table from dominating the parameter count, ensuring nearly 80% of the model's 2.87M parameters are dedicated to actual transformer compute logic.
- Extreme Overtraining: Trained on 20 Billion tokens, resulting in a ~1:7,000 parameter-to-token ratio. This forces the small architecture to memorize syntax and knowledge far beyond Chinchilla-optimal limits.
- Data Mixture:
- 60% Finephrase (Synthetic FineWeb-Edu)
- 20% DCLM (Web text)
- 10% FineMath (Mathematics)
- 10% CornStack (Code)
- Optimizer: Utilizes a hybrid Muon (for 2D weight matrices) and AdamW (for embeddings/norms) optimizer setup for stable, rapid convergence.
Benchmark Results (Open SLM Leaderboard)
| Benchmark | Score (acc_norm) |
|---|---|
| HellaSwag | 27.35% |
| ARC-Easy | 33.25% |
| ARC-Challenge | 21.08% |
| PIQA | 54.41% |
| ArithMark-3 | 32.90% |
| Intelligence Index | 5.94 |
We used the Language Model Evaluation Harness for Hellaswag, ARC-Easy, ARC-Challange, PIQA. We used the provided script for ArithMark-3
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "SurjoLabs/Ember"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgement
We would like to thank AxiomicLabs for proving that XSA architecture is excellent for token efficiency.
Limitations
This is an early test of the larger Surjo Project. The code is not very stable and we do not recommend using the modeling file without doing edits for training your own model.