Mini-K3-1H-llama-mha-block4
This repository contains a text-only Mini-K3-1H Llama-attention control from a four-model MHA/GQA by residual-mechanism comparison. It retains K3's Stable LatentMoE, first dense SiTU-GLU layer, tokenizer, and online Quantile Balancing, while all 13 sequence mixers use ordinary bias-free Q/K/V/O attention with full-head RoPE. It is a research control and is not an official Meta Llama model.
Architecture
- Decoder layers: 13, all standard MHA
- Hidden width / Q heads / KV heads / head width: 1024 / 8 / 8 / 128
- Position encoding: full-head RoPE, theta 10000.0, reset at packed-document boundaries
- Attention projections: bias-free Q/K/V/O; no MLA compression and no attention output gate
- Residual mechanism: Block-4 Attention Residuals
- Dense layers before MoE: 1
- Routed/shared experts and top-k: 64 / 2 / 4
- Routed expert hidden width: 512
- Logical parameters: 979,984,128
- Activated parameters per token: 316,759,808
- Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839
Checkpoint
- Revision/tag:
checkpoint-tokens-016000000000-final - Valid next-token targets consumed: 16,000,000,000
- Optimizer steps completed: 24,415
- Sequence length: 8,192
- Parameter dtype: BF16, with normalization and router control state retained in FP32 where defined by the implementation
- Optimizer state: deliberately not published
Every numbered checkpoint is an immutable Git tag. main points to the most
recent uploaded checkpoint. The final tag is exactly
checkpoint-tokens-016000000000-final and is created only after exactly
16,000,000,000 valid loss targets have been processed.
Initialization and comparability
All four runs use canonical name-and-shape keyed initialization with base seed
20260914. Parameters shared by two architectures, with the same semantic name
and shape, therefore begin byte-identically. Variant-only or shape-changing
parameters receive their own deterministic streams. All runs consume the same
immutable mixture schedule in the same order. Packed documents are hard-isolated
by Transformer Engine variable-length THD attention, and RoPE positions restart
from zero at every document boundary.
Training recipe
Training uses K3-style Per-Head Muon for Q/K/V matrices, Muon for other matrix parameters, AdamW fallback for vectors and embeddings, weight decay 0.1, GQA-aware per-head QK-Clip, cosine decay, 1% linear warmup, and online 1,000-bin histogram Quantile Balancing. MHA has eight Q/K/V Muon blocks; GQA has eight Q blocks and four K/V blocks. The router selects experts with biased scores and combines them with renormalized unbiased sigmoid scores. No post-training was performed.
The exact frozen source revisions, token quotas, schedule hashes, optimizer configuration, hardware benchmark choice, and validation split hashes are in the JSON manifests included in this repository. Source datasets retain their own licenses and terms; this repository does not redistribute their text.
Files and loading
model.safetensors uses the standalone layout defined by
modeling_mini_k3.py and configuration_mini_k3.py. The bundled code and
config.json fully define the architecture without relying on the original
training checkout. ARCHITECTURE.md and VARIANT.md describe the controlled
change. See initialize_model.py and smoke_test.py for local use.
Evaluation
This is an intermediate pretraining research checkpoint. It has not yet been evaluated on downstream tasks. Training-time fixed-development NLL and perplexity are logged in W&B and the run's JSONL metrics. Planned downstream and architecture diagnostics are documented in the experiment repository.
Limitations
The model is a small research proxy, is pretrained only, and should not be treated as an instruction-following assistant. Outputs may be inaccurate, biased, unsafe, or repetitive. Architecture rankings at this scale and at an 8K training length need confirmation before extrapolation to full Kimi-K3.