Summer-0.5B-S0

Summer-0.5B-S0 is a from-scratch bilingual (Chinese/English) base model: 524,336,128 parameters, trained from random initialization on 14.6B tokens with a self-trained 81,903-piece tokenizer.

It is not a fine-tune or a tokenizer-swap of any existing model. The architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 / GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight starts from N(0, 0.02).

Stage

S0 — monolingual only, from-scratch pretraining. 14.6B tokens, English 70% / Chinese 30%, 55,694 steps.

This is a retrain of the original Summer-0.5B-S0, redesigned to align with nanochat's actual training recipe:

• English is a single source (FineWebEdu, HuggingFaceFW/fineweb-edu), matching nanochat's karpathy/fineweb-edu-100b-shuffle — not the five-source blend the original release used.
• Chinese is still multiple sources, but screened first: two sources found to be 51.8% / 55.7% Traditional-Chinese-dominant by sampled character frequency (against a downstream instruction mix that is 100% Simplified) were dropped rather than mixed in.
• Packing is BOS-aligned best-fit (bos_bestfit): every training row starts with <bos> and packs whole documents greedily by best fit, cropping only the one document (if any) that doesn't fit the remainder — the same algorithm as nanochat's tokenizing_distributed_data_loader_with_state_bos_bestfit, verified line-by-line against nanochat's source. The original release used a continuous stream with no BOS token at all.
• Sequence length is 2048, matching nanochat (max_seq_len=2048 since its first commit, used unchanged at every depth from d4 to d26+).

Every input to this model must start with <bos>. It has never seen a sequence that doesn't. example_load.py / example_vllm.py in this repo do this for you — if you tokenize text yourself without prepending <bos>, the model will produce degenerate repetitive output regardless of prompt.

No parallel or instruction data at any point.

What to expect

This is a 14.6B-token model. For scale: Qwen3-0.6B-Base saw 36T tokens — about 2,700x more. Treat the numbers below as what that budget buys, not as a competitive result.

Letter multiple-choice (nanochat's primary format — render the question, model answers a single letter)

This release scores above the random baseline (0.25) on all four tasks; the previous release was below it on three of four.

Likelihood-scoring protocol (lm-evaluation-harness convention)

WMT22 5-shot translation

Few-shot translation is still essentially zero — expected, this checkpoint has never seen parallel text. The model produces fluent but off-topic continuations rather than translations; it does not yet follow the in-context examples. See Summer-0.5B-S1 for the version where in-context translation appears. Its value is as (a) a from-scratch bilingual base model in its own right, and (b) a starting point for annealing / SFT.

Tokenizer

The tokenizer is a compiled C++ extension, not loadable by AutoTokenizer. The release ships tokenizer.py and example_load.py:

pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py

The model code (model.py, checkpoint.py) is bundled too — the package depends only on torch plus the tokenizer extension, not on transformers.

Training

Full pipeline, data mixes and every design decision (including the mistakes) are documented in https://github.com/Ismantic/Summer. Notably docs/WHY.md records why fp32 master weights are mandatory, why the learning rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia, CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream). Please observe their respective licenses.