MoonViT-V2
MoonViT-V2 is the native-resolution vision backbone of Kimi K3.
| Attribute | Specification |
|---|---|
| Parameters | ~0.4B |
| Architecture | 27-layer ViT · 1024 hidden · 12 heads · 1536 QKV dim |
| Patch Size | 14 × 14 |
| Token Compression | 2×2 pixel-shuffle + temporal pooling |
| Max Resolution | ~3584 × 3584 |
| Initialization | From scratch (not SigLIP) |
| Source Model | moonshotai/Kimi-K3 |
| Technical Report | Kimi K3 (arXiv:2607.24653) |
Overview
MoonViT-V2 encodes images and videos into visual tokens for Kimi K3’s shared multimodal embedding space (via a lightweight MLP projector in the full checkpoint).
Unlike MoonViT-SO-400M (SigLIP-initialized), MoonViT-V2 is randomly initialized and jointly trained with the language model under next-token prediction.
- Unified image/video path — shared parameters with factorized spatial (intra-frame) and temporal (inter-frame) attention.
- Token compression — 2×2 pixel-shuffle (~4× fewer spatial tokens) plus temporal pooling, for high-res images and long videos within Kimi K3’s up-to-1M-token context.
- Training stability — RMSNorm and bias-free linear/attention projections; from-scratch training avoided gradient spikes seen with contrastive vision initializations at this scale.
Usage
import torch
from transformers import AutoModel, AutoImageProcessor
model_id = "AI4Industry/MoonViT-V2"
model = AutoModel.from_pretrained(
model_id,
dtype=torch.bfloat16,
trust_remote_code=True,
)
processor = AutoImageProcessor.from_pretrained(model_id, trust_remote_code=True)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
Image input
from PIL import Image
image = Image.open("your_image.png").convert("RGB")
inputs = processor(image, return_tensors="pt") # T=1
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device)
with torch.no_grad():
image_features = model(pixel_values, grid_thws)
print(image_features[0].dtype, image_features[0].shape)
# e.g. torch.bfloat16, torch.Size([N, 4, 1024])
Video input
Pass an ordered list of frames. All frames share the first frame’s resize/pad so spatial grids align. T must be ≤ max_num_frames (default 4, matching temporal position embedding).
from PIL import Image
frames = [
Image.open("frame_0.png").convert("RGB"),
Image.open("frame_1.png").convert("RGB"),
Image.open("frame_2.png").convert("RGB"),
Image.open("frame_3.png").convert("RGB"),
]
inputs = processor.preprocess_video(frames, return_tensors="pt") # T=len(frames)
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device) # [[T, H_patches, W_patches]]
with torch.no_grad():
video_features = model(pixel_values, grid_thws)
print(grid_thws.tolist(), video_features[0].shape)
# e.g. [[4, H, W]], torch.Size([N, 4, 1024]) # N after temporal pool + 2×2 merge
Notes
pixel_values: packed patches(num_patches, 3, 14, 14)grid_thws:(batch, 3)=(T, H_patches, W_patches); images useT=1- Normalization: mean/std
(0.5, 0.5, 0.5) - Optional:
attn_implementation="flash_attention_2"if FlashAttention-2 is installed (default is SDPA)
Weight Provenance
Weights (~401M params, bfloat16) are taken from model-00096-of-000096.safetensors in moonshotai/Kimi-K3.
| Original Kimi K3 Prefix | This Repo |
|---|---|
vision_tower.* | * (prefix stripped) |
mm_projector.* | omitted |
language_model.* | omitted |
Training Data & Infrastructure
Per the technical report, pretraining covers captions, interleaved documents, OCR, video, visual code, grounding (absolute + normalized coords), SVG, 3D assets, web pages, games, and CAD. Training uses context-parallel KV gather, image sharding across devices, and hiding ViT compute in pipeline bubbles. Typical use cases: OCR, visual UI iteration, and video understanding.
License
Inherits the Kimi K3 License