Flare (130M Parameters)
Flare is a highly efficient small language model developed by SurjoLabs, demonstrating that this custom XSA recurrent architecture is viable and can yield exceptional reasoning capabilities in tiny architectures.
Methodology
Flare is built to prove that unique recurrent architectures can match or beat standard transformers even at small scales.
- Custom Architecture: Built on a Llama-variant framework but features XSA Attention (value-subtraction projection) and Recurrent Layers. By reusing weights across recurrent passes, the model achieves extended effective depth while carrying a reduced set of unique weights.
- Optimized Tokenizer: Uses a custom 32,768-vocabulary (2^15) English tokenizer. This power-of-two vocabulary optimizes tensor parallelism and compute alignment while providing strong compression rates across text, code, and math.
- Extreme Overtraining: Trained on 26.21 Billion tokens (25,000 steps at 1,048,576 tokens per step), resulting in a ~1:200 parameter-to-token ratio. This forces the architecture to memorize syntax and knowledge far beyond Chinchilla-optimal limits.
- Data Mixture:
- 60% Finephrase (Synthetic FineWeb-Edu)
- 20% DCLM (Web text)
- 10% FineMath (Mathematics)
- 10% CornStack (Code)
- Optimizer: Utilizes a hybrid Muon (for 2D weight matrices) and AdamW (for embeddings/norms) optimizer setup for stable, rapid convergence.
Benchmark Results (Open SLM Leaderboard)
Evaluated at checkpoint-24999.
| Benchmark | Score (acc_norm) |
|---|---|
| HellaSwag | 39.43% |
| ARC-Easy | 51.14% |
| ARC-Challenge | 26.88% |
| PIQA | 65.94% |
| ArithMark-3 | 39.20% |
| Intelligence Index | 22.50 |
We used the Language Model Evaluation Harness for HellaSwag, ARC-Easy, ARC-Challenge, and PIQA (0-shot).
We used the provided script for ArithMark-3.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "SurjoLabs/Flare"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgement
We would like to thank AxiomicLabs for proving that XSA architecture is excellent for token efficiency.
Limitations
This is an early test of the larger Surjo Project. The code is not very stable and we do not recommend using the modeling file without doing edits for training your own model.