Sky-3B-SORE (Fine-Tuned)
Spectral Orthogonal Retrofit Enhancement (SORE) -- Fine-Tuned Variant
0labs | Sapient Intelligence | May 2026
Model Description
Sky-3B-SORE is a 3B-parameter language model created by applying the SORE (Spectral Orthogonal Retrofit Enhancement) framework to the sapientinc/HRM-Text-1B base model, followed by instruction fine-tuning on just 4.5M tokens.
SORE is a novel post-hoc architectural enhancement method that injects advanced reasoning and instruction-following capabilities into pretrained language models using SVD-based spectral decomposition -- requiring up to 10,000x less training compute and 1,000x less data than comparable models.
Key Results
| Benchmark | Base Model (1.18B) | Fine-Tuned SORE (2.99B) | Delta |
|---|---|---|---|
| ARC-Challenge (25-shot) | 81.9% | 84.4% | +2.5% |
| MATH (0-shot CoT) | 56.2% | 65.0% | +8.8% |
| Instruction Following | 33.3% | 98.0% | +64.7% |
| Overall Capabilities | 35.7% | 57.1% | +21.4% |
| MMLU (5-shot) | 60.8% | 61.0% | +0.2% |
Comparison vs. Larger Models
| Model | Parameters | Avg Benchmark | Training Data |
|---|---|---|---|
| Sky-3B-SORE (Ours) | 3B | 70.3% | 40B tokens |
| GPT-3.5 | ~175B | 63.4% | 30T tokens |
| Llama 3.1 8B | 8B | 66.1% | 15T tokens |
| Gemma 2 9B | 9B | 61.4% | 8T tokens |
| Olmo 3 7B | 7B | 64.7% | 6T tokens |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"0labs-in/Sky-3B-SORE",
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("0labs-in/Sky-3B-SORE")
prompt = "You are a helpful assistant.\n\nUser: What is the capital of France?\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
@misc{jadav2026sore,
title={SORE: Spectral Orthogonal Retrofit Enhancement for Post-Hoc Model Enhancement},
author={Atharvsinh Jadav},
year={2026},
organization={0labs / Sapient Intelligence}
}
License
Apache 2.0