Mini-K3-1H-mamba2-n64-g8
This is a text-only Mini-K3-1H research checkpoint. Nine KDA sequence mixers have been replaced with Mamba-2/SSD mixers. The four NoPE Gated MLA layers, Stable LatentMoE, Block-4 Attention Residuals, tokenizer, and untied language model head retain the baseline Mini-K3-1H design. This is an ablation within the Mini-K3 backbone, not a reproduction of NVIDIA Nemotron's block schedule.
Architecture
- Hidden width / decoder layers: 1024 / 13
- Mamba-2 layer indices:
[1, 2, 3, 5, 6, 7, 9, 10, 11]; Gated MLA layer indices:[4, 8, 12, 13] - Mamba-2: expansion 2, 32 heads × 64 channels, state dimension 64, B/C groups 8, causal depthwise-convolution kernel 4
- MLA positional mode: NoPE; output gate retained
- Routed / shared experts, top-k: 64 / 2 / 4
- Logical / project-defined active parameters: 1,009,273,696 / 346,049,376
- Vocabulary / BOS / EOS / PAD: 163840 / 163584 / 163586 / 163839
Checkpoint and training
- Immutable revision:
checkpoint-tokens-016000000000-final - Valid next-token targets consumed: 16,000,000,000
- Optimizer steps: 24,415
- Training sequence length: 8,192; local/global batch: 5/80
- Main parameter dtype: BF16; Mamba control parameters and router controls: FP32 where defined
- Optimizer state: not published
Every run begins with a randomly initialized checkpoint at token zero. The
intermediate checkpoints are at 0.5B, 1B, 2B, 4B, 8B, and 12B valid targets.
The final checkpoint tag is checkpoint-tokens-016000000000-final, created
only after exactly 16,000,000,000 targets and final validation. No
post-training is included.
All four Mamba variants and the Mini-K3-1H reference use the same frozen
tokenized mixture schedule, data order, seed, and common parameter
initialization rule. Shared parameters with identical semantic names and
shapes start identically. Mamba's new joint projections and state controls
are independently but deterministically initialized. The optimizer uses
Muon for matrix parameters and AdamW fallback; Mamba A_log, D, and
dt_bias are exempt from weight decay, as recorded in the recipe.
Packed documents are causally isolated. MLA uses document-specific causal
attention. Mamba-2 receives the same indexed document IDs in both its causal
convolution and SSD scan, which resets convolution history and recurrent
state flow at every document boundary. The gate verifies that a loss on a
later document produces exactly zero gradient on earlier-document inputs.
Packed and separately launched documents may differ by a few BF16 ULPs
because the fused FP32 prefix reductions use a different reduction tree; the
measured difference is bounded by 0.03125 and does not create an information
or gradient path across documents. All four variants passed multi-boundary,
multi-seed isolation tests, a full-model fused-kernel gate, and three
consecutive real 8K/B5 optimizer updates before formal training. The exact
gate records and source/recipe manifests are included in training/.
Files and loading
model.safetensors follows the standalone parameter schema in this repository.
The bundled config.json, configuration_mini_k3.py, modeling_mini_k3.py,
mamba2_mixer.py, and tokenizer files independently define the model and can
randomly initialize it without the training checkout. The portable Mamba
forward path is a differentiable reference; the separate training system
uses the pinned fused SSD kernel after validation.
Training-time NLL/perplexity and source-level diagnostics are in the run logs. This is a small pretrained research proxy, not an instruction-tuned assistant. Results at this size, data mix, and 8K row length need confirmation before extrapolating to full Kimi-K3 or other Mamba systems.