Mini-K3-1H-attnres-block2

This repository contains a text-only Mini-K3-1H checkpoint from the controlled Attention Residuals follow-up study. The comparison includes standard PreNorm, three block granularities, and full sublayer-level AttnRes. The original Block-4 baseline is the separately published nkkbr/Mini-K3-1H-v2; this model's only architectural change is the depth-wise residual mechanism.

Architecture

• Decoder layers: 13 (9 KDA + 4 Gated MLA)
• KDA layer indices: [1, 2, 3, 5, 6, 7, 9, 10, 11]
• Gated MLA layer indices: [4, 8, 12, 13]
• Hidden width / attention heads / KDA head width: 1024 / 12 / 128
• KDA causal depthwise-convolution kernel: 4
• KDA decay groups per head: 128 (contiguous)
• MLA positional mode: NoPE; output gate: True
• Dense layers before MoE: 1
• Routed/shared experts and top-k: 64 / 2 / 4
• Routed expert hidden width: 512
• Residual mechanism: block attention over groups of 2 decoder layers
• Residual mode / decoder layers per block: block / 2
• Logical parameters: 1,016,780,524
• Activated parameters per token: 353,556,204
• Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839

Checkpoint

• Revision/tag: checkpoint-tokens-016000000000-final
• Valid next-token targets consumed: 16,000,000,000
• Optimizer steps completed: 24,415
• Sequence length: 8,192
• Parameter dtype: BF16, with KDA decay, convolution, normalization and router control state retained in FP32 where defined by the implementation
• Optimizer state: deliberately not published

Every numbered checkpoint is an immutable Git tag. main points to the most recent uploaded checkpoint. The final tag is exactly checkpoint-tokens-016000000000-final and is created only after exactly 16,000,000,000 valid loss targets have been processed.

Initialization and comparability

The five new runs and the existing Block-4 reference use canonical name-and-shape keyed initialization with base seed 20260914. Shared parameters with the same semantic name and shape begin byte-identically. Every run consumes the same frozen 16B-token mixture schedule in the same order. Packed documents are hard-isolated: MLA uses a document-blocked causal mask, while KDA recurrent state and Q/K/V short-convolution history reset at every segment boundary.

Training recipe

Training uses K3-style per-head Muon for head-expanded Q/K/V matrices, Muon for other matrix parameters, AdamW fallback for vectors and embeddings, weight decay 0.1, per-head QK-Clip, cosine decay, 1% linear warmup, and online 1,000-bin histogram Quantile Balancing. The router selects experts with biased scores and combines them with renormalized unbiased sigmoid scores. No post-training was performed.

The exact frozen source revisions, token quotas, schedule hashes, optimizer configuration, hardware benchmark choice, and validation split hashes are in the JSON manifests included in this repository. Source datasets retain their own licenses and terms; this repository does not redistribute their text.

Files and loading

model.safetensors uses the standalone layout defined by modeling_mini_k3.py and configuration_mini_k3.py. The bundled code and config.json fully define the architecture without relying on the original training checkout. ARCHITECTURE_PACKAGE_README.md, ARCHITECTURE.md, and VARIANT.md (when present) describe the standalone package and the controlled change. See initialize_model.py and smoke_test.py for local use.

Evaluation

This is an intermediate pretraining research checkpoint. It has not yet been evaluated on downstream tasks. Training-time fixed-development NLL and perplexity are logged in W&B and the run's JSONL metrics. Planned downstream and architecture diagnostics are documented in the experiment repository.

Limitations

The model is a small research proxy, is pretrained only, and should not be treated as an instruction-following assistant. Outputs may be inaccurate, biased, unsafe, or repetitive. Architecture rankings at this scale and at an 8K training length need confirmation before extrapolation to full Kimi-K3.