M2-2B
1.93B-parameter AwareLiquid M2 preset (2080 width × 34 layers × 16 heads),
the M2 experimental line's 2b configuration trained via m2_training.
Byte-level tokenizer (UTF-8 bytes, vocab 256).
Status at upload: research milestone — 200,000-step FINAL checkpoint (val PPL 2.4106). model.safetensors holds the 200K weights; the raw resumable PyTorch checkpoint is t2b_200k.pt.
Training status (2026-09-28): the baseline run COMPLETED at 200,000 steps,
final val PPL 2.4106. Full milestone table below; the 200K weights live in the
M2 repo (docs/RESULTS_2B_30K.md) and on the training server
(/root/M2/runs/t2b_30k.pt, 3.86GB bf16). The 200K raw checkpoint
is now in this repo as t2b_200k.pt (PyTorch bf16, weight-tied; the matching 200K-weight model.safetensors export is in this repo since 2026-09-29).
Measured
| Milestone | val PPL (byte-level, WikiText-103) |
|---|---|
| 10K steps (feasibility) | 4.53 |
| 30K steps | 2.73 |
| 60K steps | 2.556 |
| 90K steps | 2.482 |
| 150K steps | 2.413 |
| 200K steps (final, 09-28) | 2.4106 |
Convergence: 30K→60K −6.4%, 60K→90K −2.9%, 90K→150K −2.8%, 150K→200K −0.1% (saturated).
Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation). Training: bf16 model, AdamW8bit, lr 1e-4, seq 128, batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
Boundaries (honest)
- Byte-level, not a subword model — PPL is on UTF-8 byte tokens; NOT comparable to gpt2-vocab models. Serve with byte tokenization.
- Not an instruct/chat model — raw next-token pretraining only.
- Research checkpoint — bf16 safetensors (weight-tied embedding; the tied
lm_head.weightis omitted, standard HF re-tie on load). No optimizer state: not exactly resumable. - The M2 line's verdict so far is honest-null on reasoning mechanisms; this checkpoint is the language-line scaling artifact, not a capability claim.
SFT checkpoints (post-training experiments, 2026-10)
| file | recipe | val PPL | JevBench public (official harness) |
|---|---|---|---|
t2b_sft_v1.pt | alpaca-short 5.4k pairs (<=128B), 3 epochs, lr 1e-4 | 3.95 | - |
t2b_sft_v2.pt | alpaca-short512 19.6k pairs (<=512B), 1+5k steps, lr 2e-5, rope_scale 4 | 3.20 | raw 31.3% (48 easy + 72 original decisions) |
Boundaries: instruction-SFT on short pairs; held-out generation went from
0/8 (base) to 4/8 format-correct, but decision-task accuracy remains the open
problem (best family configuration: 2B frozen core + decision head, 39.6%
easy-tier raw / ECE 0.20 / 0.168 s - see github.com/AwareLiquid/O1-Flash).
Serving interface for all of the above: byte-level, max_seq_len per config.
Config
config.json carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads,
gwtb_n_heads=1, vocab 256, max_seq_len 129).