Mini-K3-1H-attn-4kda
This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding.
Architecture
- Decoder layers: 13 (12 KDA + 1 Gated MLA)
- KDA layer indices:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] - Gated MLA layer indices:
[13] - Hidden width / attention heads / KDA head width: 1024 / 12 / 128
- KDA causal depthwise-convolution kernel: 4
- KDA decay groups per head: 128 (contiguous)
- MLA positional mode: NoPE; output gate: True
- Dense layers before MoE: 1
- Routed/shared experts and top-k: 64 / 2 / 4
- Routed expert hidden width: 512
- Attention Residual block size: 4
- Logical parameters: 1,028,476,624
- Activated parameters per token: 365,252,304
- Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839
Checkpoint
- Revision/tag:
checkpoint-tokens-016000000000-final - Valid next-token targets consumed: 16,000,000,000
- Optimizer steps completed: 24,415
- Sequence length: 8,192
- Parameter dtype: BF16, with KDA decay, convolution, normalization and router control state retained in FP32 where defined by the implementation
- Optimizer state: deliberately not published
Every numbered checkpoint is an immutable Git tag. main points to the most
recent uploaded checkpoint. The final tag is exactly
checkpoint-tokens-016000000000-final and is created only after exactly
16,000,000,000 valid loss targets have been processed.
Initialization and comparability
All 20 runs use canonical name-and-shape keyed initialization with base seed
20260914. Parameters shared by two architectures, with the same semantic name
and shape, therefore begin byte-identically. Variant-only or shape-changing
parameters receive their own deterministic streams. All runs consume the same
immutable, append-only mixture schedule in the same order. Packed documents are
hard-isolated: MLA uses a document-blocked causal mask, while KDA recurrent
state and Q/K/V short-convolution history reset at every segment boundary.
Training recipe
Training uses K3-style per-head Muon for head-expanded Q/K/V matrices, Muon for other matrix parameters, AdamW fallback for vectors and embeddings, weight decay 0.1, per-head QK-Clip, cosine decay, 1% linear warmup, and online 1,000-bin histogram Quantile Balancing. The router selects experts with biased scores and combines them with renormalized unbiased sigmoid scores. No post-training was performed.
The exact frozen source revisions, token quotas, schedule hashes, optimizer configuration, hardware benchmark choice, and validation split hashes are in the JSON manifests included in this repository. Source datasets retain their own licenses and terms; this repository does not redistribute their text.
Files and loading
model.safetensors uses the standalone layout defined by
modeling_mini_k3.py and configuration_mini_k3.py. The bundled code and
config.json fully define the architecture without relying on the original
training checkout. ARCHITECTURE_PACKAGE_README.md, ARCHITECTURE.md, and
VARIANT.md (when present) describe the standalone package and the controlled
change. See initialize_model.py and smoke_test.py for local use.
Evaluation
This is an intermediate pretraining research checkpoint. It has not yet been evaluated on downstream tasks. Training-time fixed-development NLL and perplexity are logged in W&B and the run's JSONL metrics. Planned downstream and architecture diagnostics are documented in the experiment repository.
Limitations
The model is a small research proxy, is pretrained only, and should not be treated as an instruction-following assistant. Outputs may be inaccurate, biased, unsafe, or repetitive. Architecture rankings at this scale and at an 8K training length need confirmation before extrapolation to full Kimi-K3.