chrono-2021-sn38 — SN38 Round 12 (year 2021)
Continue-pretrained sn38-nanochrono (~2.02B) for Bittensor subnet 38, cutoff 2021-12-31.
- Pin (this repo):
f62e238265e3ca4a4ed20b300475513446d9e1fc= train step-4000 - Base:
anacoluthe89/chrono-2020@633eb832c3de8eceb0b5bba1c5fb94faa8e777c6(UID 131) - Data + scripts:
kepton0117/sn38-train-2021 - Code: github.com/kepton0117/sn38
- Cosine vs UID 131 at this pin: 0.9294 (copy gate needs < 0.95)
recipe/ in this repo is the train algorithm and launch scripts that produced the pin.
Algorithm
Full-parameter causal LM continue-pretrain (CPT). Not LoRA. Not instruction SFT.
loss = cross_entropy(model(tokens[:-1]), tokens[1:])
| Piece | Value |
|---|---|
| Arch | sn38-nanochrono, 28 layers, hidden 1792, seq 2048 |
| Update | all parameters |
| Replay | 15% FineWeb 2017–2020 |
| Packing | docs concatenated, cut every 2048 tokens, EOS between docs |
| Optimizer | AdamW, weight decay 0.1, betas (0.9, 0.95) |
| Schedule | cosine decay, 3% warmup |
| Parallel | 8× H200 DDP |
Recipe that produced this pin
| Knob | Value |
|---|---|
| GPUs | 8× H200 |
| Micro-batch / GPU | 16 |
| Grad accum | 2 |
| Tokens / step | 8 × 16 × 2048 × 2 = 524,288 |
| LR | 1e-3 (1e-4 / 3e-4 left cosine stuck at ~0.999) |
| Checkpoint | step-4000 |
| Data | FineWeb last-4 2021 dumps (60%) + older FineWeb (15%) + brimmann2/enwiki-dec2021 (25%) |
| Filter | drop short / nav / repetitive docs (~12.5%) |
| Packed tokens | 2,610,770 × 2048 ≈ 5.35B |
# needs sn38.architectures from the GitHub repo
LR=1e-3 bash scripts/run_train_h200x8_2021.sh
Do not use FineWeb name='default' or Wikipedia 20220301.en — those leak 2022+.
Files
| Path | What |
|---|---|
model.safetensors | step-4000 weights |
config.json | nanochrono config |
recipe/ | launcher, trainer, data build, filter |