littlerock-1M-arithmax
A study in how far a 1M-parameter model can be pushed on one benchmark, and where that pushing stops being real ability.
It is opencerebral/littlerock-1M finetuned on synthetic arithmetic word problems. It reaches 40.4% on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test — well above the 25% chance floor, so the ability is not memorized. But on their private variety set, which rephrases the same kinds of problems, it drops to 21.5%. The arithmetic is real and narrow: it survives new problems, not new wording.
| ArithMark-3 set | Score | |
|---|---|---|
| Public (published set) | 40.4% | strong for 1M parameters |
| Private held-out (memorization test) | 35.5% | passes — above the 25% floor |
| Private variety (rephrased questions) | 21.5% | fails — below the floor |
| — base model, before finetuning | 25.2% | at chance |
Scores on the two private sets were measured by AxiomicLabs, who evaluate submissions independently. On the variety-set result the finetune was declined for leaderboard ranking, correctly, and the base model was listed instead.
No benchmark data was trained on, and no checkpoint was selected on a benchmark. The full protocol is below. This model is published because the three-way split is an unusually clean public example of a generalization failure that a careful process did not catch.
Why the variety set is the interesting one
Dropping below chance is the tell. A model that had learned nothing would score 25%. Scoring 21.5% means the finetune taught surface cues strong enough that unfamiliar phrasing actively pulls the model toward wrong answers.
The finetune ran with a deliberate anti-contamination protocol, and the protocol was sound — it just could not measure the thing that went wrong:
- No ArithMark item was trained on. It has no train split; all arithmetic training data was generated from templates written for this project.
- Every training string was checked against an index of all 16,428 items across all five scoring splits — exact context, exact context+ending, and 13-gram overlap. In control tests the guard caught 200/200 real ArithMark items, 200/200 HellaSwag, 200/200 PIQA and 100/100 case/punctuation-edited variants, and it dropped 221 genuine collisions from the corpus.
- Checkpoints were selected on a held-out dev set, never on the benchmark.
- The arithmetic dev set held out entire template families, not just instances, specifically to detect memorization. It reported 36.9%.
The independent variety set reported 21.5% — a ~15-point error in the project's own estimate of its own generalization. The cause is structural: the same author wrote both the training templates and the held-out ones, so both shared number ranges, unit phrasing and sentence rhythm. A held-out split cannot detect distribution-level specialization when you wrote both sides of it.
An earlier iteration was worse: 87.9% against a same-template dev set versus 26.9% on the real benchmark, a 61-point gap. Holding out template families narrowed that considerably but did not close it.
Per-topic breakdown (public set)
The profile looks like genuine partial skill, which is why it was convincing:
| Topic | Score | Covered by the generator? |
|---|---|---|
| division | 72.2 | yes |
| multiplication | 56.8 | yes |
| subtraction | 48.7 | yes |
| addition | 39.8 | yes |
| patterns | 13.2 | no |
| fractions: counting | 24.0 | no |
| two-step add/subtract | 15.2 | yes, did not transfer |
High on covered topics, at or below chance on uncovered ones — the opposite of what test-set contamination produces. This was taken as evidence that the ability was real. It is good evidence that the ability was not copied from the test set, which turns out to be a weaker claim than it appears, and not the same as being able to do arithmetic robustly.
Takeaway
Benchmark specialization can survive a rigorous contamination audit, honest dev-set selection, and a plausible-looking capability profile. Detecting it needs evaluation data the model's author did not write. That is a good argument for private variety sets, and a caution against trusting held-out splits you built yourself.
Intended use
This is a research artifact. If you want the general-purpose model, use the base; if you want a worked example of narrow benchmark optimisation and how it was detected, this is it. Training details, architecture and limitations are otherwise identical to the base model.