Spark (7.08M Parameters)
Spark is a highly efficient, ultra-small language model developed by SurjoLabs. It demonstrates that aggressive overtraining on high-quality data combined with a custom XSA recurrent architecture can yield exceptional reasoning capabilities in tiny architectures.
Methodology
Spark is built to prove that unique recurrent architectures can match or beat standard transformers even at the smallest scales.
- Custom Architecture: Built on a Llama-variant framework but features XSA Attention (value-subtraction projection) and Recurrent Layers. By reusing weights across recurrent passes, the model achieves the depth of an 11-layer model while only carrying 8 layers of unique weights.
- Optimized Tokenizer: Uses a custom 4,096-vocabulary English tokenizer. This intentionally small vocabulary prevents the embedding table from dominating the parameter count, ensuring nearly 85% of the model's 7.08M parameters are dedicated to actual transformer compute logic.
- Extreme Overtraining: Trained on ~19.9 Billion tokens (19,000 steps × 1,048,576 tokens/step). This forces the small architecture to memorize syntax and knowledge far beyond Chinchilla-optimal limits.
- Data Mixture:
- 60% Finephrase (Synthetic FineWeb-Edu)
- 20% DCLM (Web text)
- 10% FineMath (Mathematics)
- 10% CornStack (Code)
- Optimizer: Utilizes a hybrid Muon (for 2D weight matrices) and AdamW (for embeddings/norms) optimizer setup for stable, rapid convergence.
Benchmark Results (Open SLM Leaderboard)
| Benchmark | Score (acc_norm) |
|---|---|
| HellaSwag | 28.17% |
| ARC-Easy | 35.02% |
| ARC-Challenge | 20.99% |
| PIQA | 55.55% |
| ArithMark-3 | 36.10% |
| Intelligence Index | 7.93 |
Note on Checkpoint
The model was trained for 20,000 steps, but evaluation showed that the checkpoint at step 19,000 performed better on nearly all benchmarks. The final step (20,000) was discarded, and step 19,000 is the version hosted in the root of this repository.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "SurjoLabs/Spark"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgement
We would like to thank AxiomicLabs for proving that XSA architecture is excellent for token efficiency.
Limitations
This is an early test of the larger Surjo Project. The code is not very stable and we do not recommend using the modeling file without doing edits for training your own model.