SupraNeo-4M
A 4.07M-parameter decoder-only language model built to test a single question: how much of a tiny model should actually compute anything?
Most models at this scale spend the majority of their parameters on a vocabulary lookup table. SupraNeo-4M spends 82% on the transformer stack. The design starts from a custom 4,096-token BPE tokenizer, which drops the embedding matrix to 524k parameters and frees the rest for a deep, narrow stack. The aspect ratio (d/L ≈ 13) is deliberately below what the 100M class uses — following the MobileLLM finding that optimal depth-to-width shifts toward depth as models shrink.
Architecture
| Parameters | 4,070,240 (3.41M non-embedding, 83.9%) |
| Architecture | Qwen3 (Qwen3ForCausalLM) |
| Hidden size | 160 |
| Layers | 12 |
| Attention heads | 4 (head_dim 40) |
| KV heads | 2 (GQA 2:1) |
| MLP intermediate | 432 (SwiGLU) |
| Vocabulary | 4,096 (custom BPE, ~3.2 chars/token) |
| Context length | 1,024 |
| Normalization | RMSNorm + QK-Norm |
| Embeddings | tied |
| Precision | float32 |
No custom modeling code — trust_remote_code is not required.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("DedeProGames/SupraNeo-4M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/SupraNeo-4M")
ids = tok("The main reason that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True)
print(tok.decode(out[0]))
This is a base model with no chat template. apply_chat_template will fail by design.
Training
Pre-trained from scratch on a single NVIDIA L4 over 5B tokens (~1,230 tokens per parameter).
| Data | 84% HuggingFaceFW/fineweb-edu filtered to int_score ≥ 4, 16% mlfoundations/dclm-baseline-1.0 |
| Schedule | WSD, peak LR 4e-3, 1,500-step warmup, 1-sqrt decay over the final 20% |
| Anneal | last 20% on the FineWeb-Edu subset only, context extended 512 → 1,024 |
| Batch | 65,536 tokens/step |
| Optimizer | AdamW (β 0.9/0.95, wd 0.1 on 2D params, grad clip 1.0), z-loss 1e-4 |
| Init | residual branches scaled by 1/√(2L) |
The heavy FineWeb-Edu weighting is deliberate. DCLM-baseline was tuned to win reasoning benchmarks in the 1–7B range, and none of that transfers at 4M — what a model this size can learn is register and local fluency, and FineWeb-Edu's uniform expository prose is far easier to model.
Evaluation — BananaMind Base Bench 1.1
350 items, 4-way continuation-likelihood, add_special_tokens=False, no BOS, selection by highest conditional mean log-prob, Elo by weighted MLE with a prior of 4 games at 1000. Official run: dataset checksum verified, schema verified, 0 truncated contexts, 0 truncated continuations.
| Category | Accuracy | z vs chance | Elo | Sig. |
|---|---|---|---|---|
| Language Completion | 56.0% | +5.06 | 963 | ✅ |
| Logical Reasoning | 38.0% | +2.12 | 978 | ✅ |
| World Knowledge | 34.0% | +1.47 | 820 | |
| Context Tracking | 34.0% | +1.47 | 856 | |
| Commonsense | 26.0% | +0.16 | 770 | |
| Quantitative | 24.0% | −0.16 | 834 | |
| Code Completion | 22.0% | −0.49 | 883 |
| Overall Elo | 868 |
| Chance floor | 805 |
| Above floor | +63 |
| Raw accuracy | 33.4% (95% CI 28.5–38.4%) |
| z vs chance | +3.64 — significant |
By difficulty: easy 39.3%, medium 29.1%, hard 31.9%.
Reading these numbers honestly
The aggregate is significantly above chance, but the signal is concentrated in one place. Language Completion at 56% (+5.06σ) is the only strongly separated category, and that is exactly what a 4M model should be able to do: local grammatical and register plausibility. Logical Reasoning clears the bar marginally. The remaining five categories sit within noise, and Quantitative and Code Completion land at or slightly below chance — this model has no arithmetic or code capability, and the card should not be read as claiming otherwise.
The medium/hard inversion (29.1% vs 31.9%) is noise at n≈117, not evidence that harder items are easier.
BananaMind Base Bench was calibrated for the 65M–100M+ range. At 4M, with 350 four-way items, the detection floor at 1.96σ is roughly 29.5% accuracy — most of this benchmark simply lacks resolution here. For tracking progress at this scale, bits-per-byte on held-out text and BLiMP are the metrics with actual sensitivity. HellaSwag, PIQA and ARC are not reported because they sit at chance and measure nothing.
Limitations
SupraNeo-4M produces grammatical, register-consistent English with coherence over one to two sentences and topical drift beyond that. It has no factual reliability, no arithmetic, no code ability, and no instruction following. The 4,096-token vocabulary means it compresses text ~35% less efficiently than a standard 32k tokenizer, and its outputs are not comparable to other models by raw cross-entropy — use bits-per-byte.
This is a research artifact for studying small-scale pretraining, vocabulary budgets, and data mixtures. It is not intended for deployment.
Carbon footprint
Training was estimated to emit 0.46 kg CO₂ eq. — a single L4 for the duration of the run. For reference, that is roughly the footprint of driving a passenger car about 2.5 kilometers.
by DedeProGames