littlerock-1M
A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning.
Architecturally a stock Llama, so it loads with plain transformers — no
trust_remote_code, no custom modelling file:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/littlerock-1M")
model = AutoModelForCausalLM.from_pretrained("opencerebral/littlerock-1M")
Results
lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via
AxiomicLabs' bencharithmark-3.py (float32, default acc_norm).
| Benchmark | Score |
|---|---|
| HellaSwag | 26.43 |
| ARC-Easy | 30.98 |
| ARC-Challenge | 21.76 |
| PIQA | 52.23 |
| ArithMark-3 | 25.20 |
| Avg | 32.56 |
| Intelligence Index | 2.29 |
ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic.
A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all.
Architecture
| Parameters | 1,002,720 |
| Non-embedding | 609,504 |
| Layers | 6 |
| Hidden size | 96 |
| Heads | 6 (head dim 16) |
| FFN hidden (SwiGLU) | 224 |
| Context | 512 |
| Vocab | 4,096 |
| Position encoding | RoPE |
| Norm | RMSNorm |
| Embeddings | tied |
The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer's embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level BPE with byte fallback (no UNK is possible), fitted on 400k documents sampled from this model's own pretraining mixture.
Training
| Tokens | 5,000,000,000 |
| Mixture | 40% FineWeb-Edu, 60% DCLM-baseline |
| Steps | 19,073 |
| Batch | 262,144 tokens/step |
| Optimizer | AdamW (0.9, 0.95), wd 0.1 |
| LR | 3e-3 cosine → 3e-4, 50M-token warmup |
| Precision | bf16 autocast, fp32 master weights |
| Final val loss | 3.6251 (ppl 37.6) |
| Hardware | 1x RTX 3060, 1h 49m |
Data was materialised to a fixed local binary before training (no streaming), with documents EOS-separated, and the token budget enforced exactly at 5,000,000,000.
Limitations
This is a ~1M-parameter base model. It is a next-token predictor with no instruction tuning and no alignment, it has very limited world knowledge, and it will produce fluent but frequently false text. It exists to measure what a 1M-parameter budget can do.
Sample (prompt in italics): The capital of France is a national world of fashion, a country and many other nationwide nationality. In this case, the public has been banned in the early 17th century…