MoonViT-V2 (Standalone)

Standalone extraction of the MoonViT-V2 vision encoder (and the Kimi K3 multimodal projector) from moonshotai/Kimi-K3, so the vision tower can be used without downloading the full 2.8T-parameter model.

Why this exists

Kimi K3 ships as a single 96-shard, ~1.5 TB checkpoint whose vision subsystem coexists with the language model weights. For projects that only need the vision encoder — e.g. training a small vision→LLM projector on a frozen text backbone (adapter / vision-bridge / LLaVA-style grafting) — pulling the entire K3 MoE is wasteful and, on modest GPUs, impossible. The vision subsystem lives entirely in the two tail shards of the checkpoint, so it can be extracted losslessly.

Contents

Keys are kept under their canonical Kimi prefixes (vision_tower.*, mm_projector.*) — byte-identical to the source shards, so state loaders bind with no rename logic.

Architecture (from vision_config.json)

patch_size:            14
merge_kernel_size:     [2, 2]      # 2x2 spatial merge (PatchMergerV2)
merge_type:            sd2_tpool
vt_hidden_size:        1024        # per-patch output dim
qkv_hidden_size:       1536
intermediate_size:     4096
num_hidden_layers:     27
num_attention_heads:   12
activation:            gelu_pytorch_tanh
norm:                  rmsnorm
pos_emb:               divided_fixed (init 64x64 grid, bilinear interp)
attn_implementation:   flash_attention_2
text_hidden_size:      7168        # K3 LLM width (NOT this encoder's width)
mm_projector_type:     patchmergerv2

Embedding contract (the part adapters depend on):

• Per-patch encoder output = 1024-dim.
• The 2×2 PatchMerger concatenates 4 neighbouring patches → a 4096-dim visual token (1024 * 2 * 2). This 4096 vector is the input any downstream projector consumes.
• Image-token count after merge ≈ ceil(H/28) * ceil(W/28) (28 = patch 14 × merge 2).

The bundled kimi_mm_projector is K3's own 4096 → 7168 mapping into the K3 LLM embedding space. When grafting onto a different LLM, train a new projector sized to that LLM's hidden width (e.g. 4096 → 4096 for a 4096-hidden model) — do not reuse kimi_mm_projector, which is K3-specific.

Usage

import json, torch
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

REPO = "keypa/MoonViT-V2-Standalone"
sd    = load_file(hf_hub_download(REPO, "moonvit_v2.safetensors"))   # encoder
cfg   = json.loads(hf_hub_download(REPO, "vision_config.json"))
proj  = load_file(hf_hub_download(REPO, "kimi_mm_projector.safetensors"))  # optional

# Build a MoonViT module matching cfg, then:
#   missing, unexpected = model.load_state_dict(sd, strict=True)
# Preprocess to patch-14 grid, forward, then apply the 2x2 PatchMerger reshape.
#   feats: [num_merged_tokens, 1024] -> merge -> [num_merged_tokens/4 * ... , 4096]

A reference loader module (moonvit.py) reproducing the encoder exactly is recommended before training; verify forward parity against the source repo's trust_remote_code implementation on identical inputs.

Extraction method

1. Read model.safetensors.index.json from K3 (weight_map).
2. Vision tensors (vision_tower.*, mm_projector.*) map to shards model-00095-of-000096.safetensors and model-00096-of-000096.safetensors only (~895 MB).
3. Stream-download those two shards, carve the prefixes (names unchanged), save standalone. No weight is modified, requantized, or rescaled — byte-identical to the upstream tensors.

Intended use & caveats

• Vision feature extraction and as a frozen encoder for projector/adapter training.
• This is an encoder only — it has no text head and cannot generate or answer.
• Preprocessing parity (normalization, resolution tiling) lives in K3's trust_remote_code and must be reproduced for correct results.
• Not independently benchmarked here; treat as a research artifact.

License

Upstream moonshotai/Kimi-K3 License ("kimi-k3"). This is a derivative work of the K3 weights; all K3 license conditions apply, including Moonshot AI's commercial thresholds (e.g. Model-as-a-Service revenue / monthly-active-user limits). Read the full K3 license before commercial use. This model card summarizes but does not replace that license.

Citation

@misc{moonshot2025kimik3,
  title  = {Kimi K3},
  author = {Moonshot AI},
  year   = {2026},
  url    = {https://huggingface.co/moonshotai/Kimi-K3}
}