dont use, this model is contaminated
Sol Pro is a 138,037,594-parameter language model from Sol Labs. It combines recurrent transformer blocks with tensorized n-gram memory and a 2,048-token context.
The released checkpoint scores 28.63 (not true) on the normalized Intelligence Index and 24.13 on the raw accuracy Index.
Model
| Setting | Value |
|---|---|
| Parameters | 138,037,594 |
| Context | 2,048 tokens |
| Vocabulary | 4,096-token byte-level BPE |
| Hidden width | 768 |
| Stored / effective blocks | 12 / 16 |
| Attention | 24 query heads, 8 KV heads, head dimension 32 |
| Feed-forward layers | SwiGLU, width 3,968 |
| Position encoding | RoPE, theta 20,000 |
| Memory | Rank-299 tensorized 2-, 3-, and 4-gram memory |
| Recurrence | Learned pass embeddings and loop gates |
| Embeddings | Tied input and output weights |
| Inference weights | BF16 safetensors |
Four middle blocks run twice. Causal grouped-query attention uses learned Q/K RMSNorm and XSA value-direction subtraction. The n-gram module shares token-position factors across its three orders.
Load and generate
Install the dependencies:
pip install torch safetensors tokenizers huggingface_hub
import sys
from huggingface_hub import snapshot_download
model_dir = snapshot_download(
"solintellegence/sol-pro",
allow_patterns=[
"model.safetensors", "modeling_sol_pro.py",
"config.json", "tokenizer.json",
],
)
sys.path.insert(0, model_dir)
from modeling_sol_pro import load_model, generate
model, tokenizer = load_model(model_dir)
print(generate(
model,
tokenizer,
"The best way to learn something new is",
max_new_tokens=64,
))
The repository includes standalone PyTorch model code. Generation recomputes the context at each step; KV caching is not implemented. Keep prompts and continuations within 2,048 tokens.
Evaluation
| Benchmark | Examples | Normalized accuracy | Raw accuracy |
|---|---|---|---|
| HellaSwag | 10,042 | 46.76% | 36.88% |
| ARC Easy | 2,376 | 54.08% | 55.35% |
| ARC Challenge | 1,172 | 32.00% | 29.35% |
| PIQA | 1,838 | 70.95% | 69.26% |
| Arithmark3 | 1,000 | 36.00% | 37.20% |
| Intelligence Index | 28.63 | 24.13 |
These are zero-shot float32 measurements on complete evaluation splits. HellaSwag, ARC Easy, ARC Challenge, and PIQA use lm-evaluation-harness 0.4.12 with batch size 64. Arithmark3 uses the official AxiomicLabs script with its default batch size 32 and 1,024-token context, explicitly set to float32. The Index uses the leaderboard's chance-adjusted formula and each task's normalized accuracy.
This is a task-adapted checkpoint using public benchmark training splits. Evaluation used separate held-out splits, with matching evaluation contexts excluded from the adaptation data; Arithmark evaluation examples were not used for adaptation. The scores have not been independently verified and do not establish a leaderboard position.
WikiText-2 validation cross-entropy is 2.7677 over 366,592 tokens. Exact results, package versions, hashes, and commands are in the float32 evaluation summary. Raw outputs are available for lm-eval and ArithMark-3.
Use and limits
Sol Pro supports text-completion and small-model research. Multiple-choice scores do not establish reliable free-form reasoning or conversational behavior. Outputs can be incorrect, repetitive, or inconsistent.
Files
| File | Contents |
|---|---|
model.safetensors | Released inference weights |
modeling_sol_pro.py | Architecture, loader, and text generator |
config.json | Architecture configuration |
tokenizer.json | Original tokenizer |
evaluation/standard_float32/ | Standard-harness evaluation results and reproducibility details |
banner.png | Sol Pro artwork |
License
Sol Pro is released under Apache 2.0. Dataset licenses and terms remain with their respective owners.