Mini-K3-1H-mamba2-n128-g8

This is a text-only Mini-K3-1H research checkpoint. Nine KDA sequence mixers have been replaced with Mamba-2/SSD mixers. The four NoPE Gated MLA layers, Stable LatentMoE, Block-4 Attention Residuals, tokenizer, and untied language model head retain the baseline Mini-K3-1H design. This is an ablation within the Mini-K3 backbone, not a reproduction of NVIDIA Nemotron's block schedule.

Architecture

• Hidden width / decoder layers: 1024 / 13
• Mamba-2 layer indices: [1, 2, 3, 5, 6, 7, 9, 10, 11]; Gated MLA layer indices: [4, 8, 12, 13]
• Mamba-2: expansion 2, 32 heads × 64 channels, state dimension 128, B/C groups 8, causal depthwise-convolution kernel 4
• MLA positional mode: NoPE; output gate retained
• Routed / shared experts, top-k: 64 / 2 / 4
• Logical / project-defined active parameters: 1,018,756,960 / 355,532,640
• Vocabulary / BOS / EOS / PAD: 163840 / 163584 / 163586 / 163839

Checkpoint and training

• Immutable revision: checkpoint-tokens-016000000000-final
• Valid next-token targets consumed: 16,000,000,000
• Optimizer steps: 24,415
• Training sequence length: 8,192; local/global batch: 5/80
• Main parameter dtype: BF16; Mamba control parameters and router controls: FP32 where defined
• Optimizer state: not published

Every run begins with a randomly initialized checkpoint at token zero. The intermediate checkpoints are at 0.5B, 1B, 2B, 4B, 8B, and 12B valid targets. The final checkpoint tag is checkpoint-tokens-016000000000-final, created only after exactly 16,000,000,000 targets and final validation. No post-training is included.

All four Mamba variants and the Mini-K3-1H reference use the same frozen tokenized mixture schedule, data order, seed, and common parameter initialization rule. Shared parameters with identical semantic names and shapes start identically. Mamba's new joint projections and state controls are independently but deterministically initialized. The optimizer uses Muon for matrix parameters and AdamW fallback; Mamba A_log, D, and dt_bias are exempt from weight decay, as recorded in the recipe.

Packed documents are causally isolated. MLA uses document-specific causal attention. Mamba-2 receives the same indexed document IDs in both its causal convolution and SSD scan, which resets convolution history and recurrent state flow at every document boundary. The gate verifies that a loss on a later document produces exactly zero gradient on earlier-document inputs. Packed and separately launched documents may differ by a few BF16 ULPs because the fused FP32 prefix reductions use a different reduction tree; the measured difference is bounded by 0.03125 and does not create an information or gradient path across documents. All four variants passed multi-boundary, multi-seed isolation tests, a full-model fused-kernel gate, and three consecutive real 8K/B5 optimizer updates before formal training. The exact gate records and source/recipe manifests are included in training/.

Files and loading

model.safetensors follows the standalone parameter schema in this repository. The bundled config.json, configuration_mini_k3.py, modeling_mini_k3.py, mamba2_mixer.py, and tokenizer files independently define the model and can randomly initialize it without the training checkout. The portable Mamba forward path is a differentiable reference; the separate training system uses the pinned fused SSD kernel after validation.

Training-time NLL/perplexity and source-level diagnostics are in the run logs. This is a small pretrained research proxy, not an instruction-tuned assistant. Results at this size, data mix, and 8K row length need confirmation before extrapolating to full Kimi-K3 or other Mamba systems.