BananaMind 2.1 Lite
BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class.
- 19,950,029 parameters in the tied-token Transformer core
- 4,999,970 parameters in a two-hash causal trigram module
- BananaMind 2 Nano's 8,192-token tokenizer
- 13 physical layers and 15 effective layer passes
- execution:
L1 → L2 → L3 → L4 → L5 → L5 → L6 → L7 → L8 → L9 → L9 → L10 → L11 → L12 → L13 - both visits to L5 share one physical layer; both visits to L9 share another
- 4,096-token context
The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9. L5 and L9 have separate learned injection scales, both initialized to 0.5.
75B-token streamed curriculum
| Source | Tokens | Share |
|---|---|---|
| FineWeb-Edu | 37.50B | 50% |
| DCLM Baseline | 16.50B | 22% |
| Cosmopedia v2 | 7.50B | 10% |
| FineMath 4+ | 6.75B | 9% |
| FinePhrase | 4.50B | 6% |
| NPset-2 Python-Edu | 2.25B | 3% |
The NPset allocation is 75% normalized code and 25% raw
original_code. Curriculum weights move from web-heavy foundations to a
balanced phase and then greater math, synthetic-text, phrase, and code
coverage. A token-credit scheduler keeps the complete run within one global
batch of every target allocation.
Training uses Muon for matrix parameters, AdamW for embeddings and control parameters, and an independent AdamW learning rate for the n-gram module. Checkpoints are uploaded every 5% with safetensors, tokenizer files, pinned dataset revisions, metrics, exact source-token accounting, and complete state for resuming.
Launch on Hugging Face Jobs
# Start a new 4 x H200 run (32 sequences/GPU, global batch 128).
./launch_lite_25m_hf_job.sh 4 fresh
# Resume it later.
./launch_lite_25m_hf_job.sh 4 resume
# The same global batch on 8 x H200 (16 sequences/GPU).
./launch_lite_25m_hf_job.sh 8 resume
The launcher expects HF_TOKEN to be available to the Hugging Face CLI and
as a Hugging Face Jobs secret.