Surjo-100m is a 97.7M-parameter hybrid recurrent language model developed by SurjoLabs. Built on the Gen-3 Surjo architecture, it couples Gated DeltaNet-2 (GDN-2) linear attention with full attention (XSA).
By executing a weight-tied recurrent core across 2 passes, the model reaches an effective computational depth of 34 layers while retaining the memory footprint and latency profile of an 18-layer physical model.
Model Specifications
| Attribute | Specification |
|---|---|
| Parameters | 97.7M |
| Architecture | Hybrid GDN-2 + XSA (Subtractive Attention) |
| Physical Layers | 18 (1 Prelude + 16 Recurrent Trunk + 1 Coda) |
| Effective Depth | 34 Layers (2 Recurrent Passes) |
| Hidden Size | 576 |
| Intermediate Size | 1536 |
| Attention Configuration | GQA (9 Query heads, 3 KV heads; Head Dim: 64) |
| Linear Attention | GDN-2 (K: 64, V: 64, 9 V-Heads) |
| Context Length | 1024 tokens pretraining (2048 max) |
| Vocabulary | 32,768 (Custom BPE, tied embeddings) |
Evaluation Results
| Benchmark | Score |
|---|---|
| PIQA | 63.87% |
| ARC-Easy | 47.64% |
| ArithMark-3 | 38.90% |
| HellaSwag | 35.05% |
| ARC-Challenge | 25.85% |
| Int Index | 18.86 |
Architecture Details
Surjo Gen-3 combines fast recurrent sequence processing with full-attention boundary layers:
- Prelude (Layer 0): Standard attention with subtractive value projection (XSA) to anchor context.
- Recurrent Core (Layers 1–16, tied 2× passes): Four interleaved groups consisting of 3× GDN-2 linear attention layers followed by 1× XSA full attention layer. GDN-2 uses chunked/fused recurrent kernels via Flash Linear Attention (FLA).
- Coda (Layer 17): Terminal full-attention XSA layer.
- Tied Head: The output projection shares weights with the 32,768-token embedding layer.
Pretraining
- Dataset (20B tokens):
- 60% FinePhrase
- 20% DCLM
- 10% FineMath
- 10% Cornstack Python
- Training Schedule: 20,000 steps (~1M tokens/step) at a sequence length of 1024 tokens.
- Optimizer: Hybrid Muon (
peak lr = 0.02) and AdamW (peak lr = 0.0036).
Usage
Requirements
pip install torch transformers accelerate
pip install -U git+https://github.com/fla-org/flash-linear-attention
Generation Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SurjoLabs/Surjo-100m"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
device_map="auto",
trust_remote_code=True,
)
prompt = "The secret to scaling small language models is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation & License
This project is released under the Apache 2.0 License.
@software{surjo100m2026,
author = {SurjoLabs},
title = {Surjo-100m: Hybrid Recurrent Language Model},
year = {2026},
url = {https://huggingface.co/SurjoLabs/Surjo-100m}
}