Mini-K3-1H-attnres-full-sublayer
This repository contains a text-only Mini-K3-1H checkpoint from the controlled
Attention Residuals follow-up study. The comparison includes standard PreNorm,
three block granularities, and full sublayer-level AttnRes. The original
Block-4 baseline is the separately published nkkbr/Mini-K3-1H-v2; this
model's only architectural change is the depth-wise residual mechanism.
Architecture
- Decoder layers: 13 (9 KDA + 4 Gated MLA)
- KDA layer indices:
[1, 2, 3, 5, 6, 7, 9, 10, 11] - Gated MLA layer indices:
[4, 8, 12, 13] - Hidden width / attention heads / KDA head width: 1024 / 12 / 128
- KDA causal depthwise-convolution kernel: 4
- KDA decay groups per head: 128 (contiguous)
- MLA positional mode: NoPE; output gate: True
- Dense layers before MoE: 1
- Routed/shared experts and top-k: 64 / 2 / 4
- Routed expert hidden width: 512
- Residual mechanism: full attention over individual attention and FFN outputs
- Residual mode / decoder layers per block: full_sublayer / None
- Logical parameters: 1,016,780,524
- Activated parameters per token: 353,556,204
- Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839
Checkpoint
- Revision/tag:
checkpoint-tokens-016000000000-final - Valid next-token targets consumed: 16,000,000,000
- Optimizer steps completed: 24,415
- Sequence length: 8,192
- Parameter dtype: BF16, with KDA decay, convolution, normalization and router control state retained in FP32 where defined by the implementation
- Optimizer state: deliberately not published
Every numbered checkpoint is an immutable Git tag. main points to the most
recent uploaded checkpoint. The final tag is exactly
checkpoint-tokens-016000000000-final and is created only after exactly
16,000,000,000 valid loss targets have been processed.
Initialization and comparability
The five new runs and the existing Block-4 reference use canonical
name-and-shape keyed initialization with base seed 20260914. Shared
parameters with the same semantic name and shape begin byte-identically.
Every run consumes the same frozen 16B-token mixture schedule in the same order. Packed documents are
hard-isolated: MLA uses a document-blocked causal mask, while KDA recurrent
state and Q/K/V short-convolution history reset at every segment boundary.
Training recipe
Training uses K3-style per-head Muon for head-expanded Q/K/V matrices, Muon for other matrix parameters, AdamW fallback for vectors and embeddings, weight decay 0.1, per-head QK-Clip, cosine decay, 1% linear warmup, and online 1,000-bin histogram Quantile Balancing. The router selects experts with biased scores and combines them with renormalized unbiased sigmoid scores. No post-training was performed.
The exact frozen source revisions, token quotas, schedule hashes, optimizer configuration, hardware benchmark choice, and validation split hashes are in the JSON manifests included in this repository. Source datasets retain their own licenses and terms; this repository does not redistribute their text.
Files and loading
model.safetensors uses the standalone layout defined by
modeling_mini_k3.py and configuration_mini_k3.py. The bundled code and
config.json fully define the architecture without relying on the original
training checkout. ARCHITECTURE_PACKAGE_README.md, ARCHITECTURE.md, and
VARIANT.md (when present) describe the standalone package and the controlled
change. See initialize_model.py and smoke_test.py for local use.
Evaluation
This is an intermediate pretraining research checkpoint. It has not yet been evaluated on downstream tasks. Training-time fixed-development NLL and perplexity are logged in W&B and the run's JSONL metrics. Planned downstream and architecture diagnostics are documented in the experiment repository.
Limitations
The model is a small research proxy, is pretrained only, and should not be treated as an instruction-following assistant. Outputs may be inaccurate, biased, unsafe, or repetitive. Architecture rankings at this scale and at an 8K training length need confirmation before extrapolation to full Kimi-K3.