BananaMind 2.1 Lite
AdamW training checkpoints are published separately at
Banaxi-Tech/BananaMind-2.1-Lite-AdamW so the earlier Muon experiment is
preserved.
BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class.
- 19,950,029 parameters in the tied-token Transformer core
- 4,999,970 parameters in a two-hash causal trigram module
- BananaMind 2 Nano's 8,192-token tokenizer
- 13 physical layers and 15 effective layer passes
- execution:
L1 → L2 → L3 → L4 → L5 → L5 → L6 → L7 → L8 → L9 → L9 → L10 → L11 → L12 → L13 - both visits to L5 share one physical layer; both visits to L9 share another
- 4,096-token context
The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9. L5 and L9 have separate learned injection scales, both initialized to 0.5.
75B-token streamed curriculum
| Source | Tokens | Share |
|---|---|---|
| FineWeb-Edu | 37.50B | 50% |
| DCLM Baseline | 16.50B | 22% |
| Cosmopedia v2 | 7.50B | 10% |
| FineMath 4+ | 6.75B | 9% |
| FinePhrase | 4.50B | 6% |
| NPset-2 Python-Edu | 2.25B | 3% |
The NPset allocation is 75% normalized code and 25% raw
original_code. Curriculum weights move from web-heavy foundations to a
balanced phase and then greater math, synthetic-text, phrase, and code
coverage. A token-credit scheduler keeps the complete run within one global
batch of every target allocation.
Training uses pure AdamW with (0.9, 0.95) betas: peak LR 0.003 for the
Transformer core, embeddings, and controls, and peak LR 0.001 for the n-gram
module. Both rates use warmup-stable-decay with a final 15% cosine decay.
Checkpoints are uploaded every 5% with safetensors, tokenizer files, pinned
dataset revisions, metrics, exact source-token accounting, and complete state
for resuming.
Launch on Hugging Face Jobs
# Start a new 4 x H200 run (32 sequences/GPU, global batch 128).
./launch_lite_25m_hf_job.sh 4 fresh
# Resume it later.
./launch_lite_25m_hf_job.sh 4 resume
# The same global batch on 8 x H200 (16 sequences/GPU).
./launch_lite_25m_hf_job.sh 8 resume
The launcher expects HF_TOKEN to be available to the Hugging Face CLI and
as a Hugging Face Jobs secret.