Summer-0.5B-S1
Summer-0.5B-S1 is a from-scratch bilingual (Chinese/English) base model: 524,336,128 parameters, trained from random initialization on 14.6B + 1.34B anneal tokens with a self-trained 81,903-piece tokenizer.
It is not a fine-tune or a tokenizer-swap of any existing model. The
architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 /
GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight
starts from N(0, 0.02).
Stage
S1 — S0 plus a parallel-data anneal, same nanochat-aligned recipe as
Summer-0.5B-S0 (BOS-aligned best-fit packing, seq_len 2048). Branched from
S0's final checkpoint (optimizer state reset — verified by A/B test to
behave identically to resuming from a checkpoint with saved optimizer
momentum) and trained for 5,103 steps on 1.34B tokens containing ~30%
Chinese-English parallel text, packed the same BOS-aligned way as S0 (the
anneal data must match the pretraining packing convention, or the model sees
an out-of-distribution input shift mid-training).
Every input must start with <bos>, same as Summer-0.5B-S0 — see that
model's card for why.
What to expect
This is a 14.6B + 1.34B anneal-token model. For scale: Qwen3-0.6B-Base saw 36T tokens —
about 2,700x more. Treat the numbers below as what that budget buys, not as a
competitive result.
| WMT22 5-shot | BLEU | COMET |
|---|---|---|
| zh->en | 8.99 | 0.6883 |
| en->zh | 28.36 | 0.7736 |
Against this model's own pre-anneal state (S0, 0.29 / 2.78 BLEU) this is
qualitative: S0 ignores the in-context examples and produces off-topic
continuations, S1 actually translates. Against the previous Summer-0.5B-S1
release (8.99 / 27.29 BLEU, COMET 0.6855 / 0.7743) this new release performs
at parity — the nanochat-aligned data recipe did not cost any translation
quality while improving the base model on every other tracked metric.
Tokenizer
The tokenizer is a compiled C++ extension, not loadable by
AutoTokenizer. The release ships tokenizer.py and example_load.py:
pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py
The model code (model.py, checkpoint.py) is bundled too — the package
depends only on torch plus the tokenizer extension, not on transformers.
Training
Full pipeline, data mixes and every design decision (including the mistakes)
are documented in https://github.com/Ismantic/Summer. Notably
docs/WHY.md records why fp32 master weights are mandatory, why the learning
rate schedule is WSD rather than cosine, and what the vocabulary swap cost.
License
Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia,
CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream).
Please observe their respective licenses.