cagliostro-v3

A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the Open SLM Leaderboard metric.

The training run is complete. 75.00B tokens, 762,939 steps, learning rate decayed to zero.

Results

Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository.

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge.

For context against other models at this scale, using the leaderboard's published figures:

SmolLM2-135M is 0.58 Index ahead on roughly 27 times the training tokens. Token counts for the other models are as published by their authors.

No single benchmark carries the cooldown gain. All five move together, which is the signature of the learning rate depression lifting rather than the model acquiring something new in the last 11B tokens.

The shape of the result is lopsided. cagliostro-v3 beats both models shown on ArithMark-3 by more than 4 points, which is what tripling the mathematics share during the cooldown bought. Across the whole board it places third on that benchmark, behind MobileLLM-R1-140M-base at 65.70 and palmer-006 at 52.70. It trails on PIQA, where GPT-X2.5-135M sits at 69.42 against our 67.46, and PIQA carries the heaviest weight in the Index at 0.548 per point. That single task is most of the remaining gap to SmolLM2-135M.

The flat stretch from 30B to 64B is not a stall. A warmup-stable-decay schedule holds the learning rate at its peak for the first 85% of the run, and constant peak learning rate depresses zero-shot multiple-choice accuracy even while validation loss keeps improving. The recovery from 22.0 to 26.6 is what the cooldown unlocks, not new knowledge appearing in the last 9B tokens.

The sharp drop in training loss at 63.75B is the data mixture changing, not the model improving. Validation loss on a fixed held-out set is the honest line, and it moves smoothly.

Model details

The architecture is defined in this repository. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.

Training data

Two mixtures. The first covers the stable phase, the second takes over when the cooldown begins at 85% of the run.

Mathematics rises from 10% to 28% during the cooldown. No source exceeds 0.4 epochs across the full run, so nothing is repeated enough to memorize.

Training setup

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/cagliostro-v3"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model with no instruction tuning and no chat template. It completes text.

Reproducing the evaluation

pip install lm-eval
python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/cagliostro-v3,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 8 --device cuda:0

ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id. Evaluate in float32. A bfloat16 round trip moves logits by about 1.3e-1 at this logit cap, which is enough to change borderline multiple-choice answers, while float32 agrees with the training weights to 3.2e-05.

Provenance

This repository holds the full training history. A checkpoint was pushed every 30 minutes from the first step, giving 363 commits between 11 and 20 September 2026. Any intermediate checkpoint can be retrieved by revision, and the benchmark trajectory in the chart above can be reproduced from them.

Limitations

English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 146M parameters it confabulates freely and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.

License

Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).