ounce100m-v1
A 106,194,240-parameter decoder-only transformer trained from scratch on
999,817,216 tokens of Cion-lab/ounce100m-mix-v1, in 3814 optimiser steps on
2×Nvidia Tesla T4 inside Kaggle notebook sessions. It is the deliverable of a project whose whole point
was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to
report it honestly, including where that is near chance.
Measured facts
| Parameters | 106,194,240 (recomputed from model.safetensors's own header, not asserted) |
| Architecture | 22 layers, hidden 576, GQA 9 Q / 3 KV heads, SwiGLU 1536, RMSNorm 1e-05, RoPE θ=10000.0, tied embeddings, vocab 49,152 |
| Sequence length | 1024 — which is why ARC's 25-shot prompts are truncated from the left by the eval harness |
| Tokens consumed | 999,817,216 (99.98 % of the 1.0 B target) |
| Batch | micro 4 × accum 32 × 2 ranks = 262,144 tokens/step |
| Optimiser | adamw_torch, LR 0.0006, warmup fraction 0.02, trapezoid decay over the final 20.0 % of the schedule |
| Precision | fp16 autocast + fp32 master weights + GradScaler — asserted at the start of every session, and each session's log line precision: fp16=True bf16=False scaler=GradScaler is the evidence |
| Throughput | 12,150 tok/s in the session that produced this export |
| Attention | eager |
| Gradient checkpointing | False |
| Randomness | model seed 20260919, data seed 20260919 |
| Wall clock | 4.59 h in the session that produced this export. The run is split across sessions at checkpoint boundaries, so this is not the total training time |
| Loss | 4.43187255859375 -> 4.264701843261719 |
| Validation PPL | 75.98 on Cion-lab/ounce100m-mix-v1's held-out val/ shards (22,934,043 tokens, never trained on) |
Data
Cion-lab/ounce100m-mix-v1: 1,109,714,831 training tokens in 139 shards + 22,934,043
held-out validation tokens, cut with HuggingFaceTB/SmolLM2-135M:tokenizer.json (49,152 vocab,
Apache-2.0), from 15 public sources at ≥15
per shard. Format and loading code are in that dataset's card.
Contamination. Overlap was measured mechanically against benchmark train/validation/dev material of
the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test
splits stayed untouched until evaluation). audit.json in the dataset repo records
overlap_total = 0 over 1,073,047 documents and
21,898 held-out documents, tasks_covered = 8/8.
Building it required dropping 5878 documents
(17,366,967 tokens)
that matched reference material; the exact exclusion masks ship in filter/ so the mix is reproducible.
Evaluate it
pip install "lm-eval==0.4.13"
lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \
--tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \
--batch_size 8 --seed 42 --log_samples
No benchmark scores are published with this model. The eight-task evaluation was dropped on
2026-09-21T14:13Z by the project owner before any task produced a score (the first eval kernel died at 1.77 s
on a module-path bug and was not relaunched), so every number on this card comes from training or from the
held-out validation set - perplexity 75.98 on 1,953 windows of val/, which the model never trained on.
The protocol that was going to be used is frozen and public anyway: task ids, metric names, shot counts,
splits and chance levels were all fixed in docs/05-eval-plan.md of Cion-lab/ounce100m-code before any score existed, with the driver in
eval/run_benchmarks.py at Cion-lab/ounce100m-code. Treat the command above as a reproduction recipe, not
as a claim that these results were measured.
Files
| file | sha256 | bytes |
|---|---|---|
config.json | be63e8d2dc708f8b… | 723 |
cursor.json | c5ea3ba9293f3278… | 205 |
generation_config.json | 1dac20576af54bdd… | 219 |
model.safetensors | 669ac130a8853cc2… | 424,799,272 |
run_summary.json | d7789a4e70e296c3… | 968 |
special_tokens_map.json | e786b595b9a23148… | 831 |
tokenizer.json | 9ca9acddb6525a19… | 2,104,556 |
tokenizer_config.json | 4bb9af56a342753d… | 3,658 |
cursor.json is the data position this checkpoint corresponds to: samples consumed, step, seed, the
corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than
trusted.
Not in this model
No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of training against the ~300 B of the comparable published reference models, so expect scores at or near chance on several tasks; that is the experiment's result, not a defect to be engineered around.