MiMo-V2.6-Distill-Qwen-9B — MXFP4 GGUF

MXFP4 quantization of Xiaomi's MiMo-V2.6-Distill-Qwen-9B, packaged for llama.cpp.

What this model is

MiMo-V2.6-Distill-Qwen-9B is a 9-billion-parameter agentic vision-language model built by Xiaomi MiMo. It is a supervised fine-tune (SFT) of Qwen3.5-9B on data generated by the larger MiMo-V2.6 model family, covering four target domains:

• Software engineering / agentic coding
• General-purpose agent work (tool use, terminal, automation)
• Visual coding (understanding and writing code from screen/UI images)
• Cybersecurity

Xiaomi released this SFT checkpoint as an open starting point for research in agentic reinforcement learning — the MiMo team's stated goal is to give the community a base checkpoint rather than a closed final product.

Model tree

Qwen/Qwen3.5-9B                       (base architecture, Alibaba Qwen)
└── XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B   (SFT by Xiaomi MiMo, MIT)
    └── FreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-MXFP4-GGUF   (this repo)

Source: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B · License: MIT · Architecture: Qwen3_5ForConditionalGeneration

Architecture

• Hybrid Mamba-2 + attention: 32 layers — 24 linear-attention (Mamba-2) layers and 8 full-attention layers (full_attention_interval: 4). The Mamba layers are why 256K context is affordable in VRAM.
• Vision encoder included (333 vision tensors, patch 16, depth 27) — image and video inputs, shipped separately as mmproj.
• Context: 262,144 tokens (256K) — max_position_embeddings: 262144, inherited from Qwen3.5-9B.
• Hidden 4096, 16 heads / 4 KV heads, head_dim 256, intermediate 12288, vocab 248,320, rope θ 10M with 0.25 partial rotary, interleaved MRoPE.
• Thinking is enabled by default (source chat template) — responses begin with [Start thinking].

Reported results (from Xiaomi's model card)

Trained on a 77.4B-token SFT mixture (27.2B loss-bearing) split across code, cyber, general, and visual data. Figures above are Xiaomi's own reported numbers, not reproduced here.

Quantization

• BPW is measured against the F16 GGUF (16.69 GiB, 427 tensors), not an idealized FP32 baseline.
• Norms, embeddings, and per-layer scales stay F32 by design — MXFP4 stores weights only.
• MXFP4 = llama.cpp ftype 42 (dense, "MXFP4 dense"). This is the OCP MXFP4 format, not MXFP4_MOE. MXFP4 is portable across backends; on NVIDIA Blackwell it runs at full speed, elsewhere it is slower but still functional.
• Hugging Face's quantization auto-detect may report "unable to determine" for this file because HF's quantization enum lags llama.cpp's. The artifact is genuine MXFP4 as listed above.

Inference performance

Measured on NVIDIA GeForce RTX 5060 Ti 16 GB (Blackwell SM120), CUDA 13.2, llama.cpp build b1868-4fea119d, full GPU offload (-ngl 99), KV cache Q8_0 for both K and V.

Notes on these numbers:

• The 128K prompt figure is from a ~20-token prompt, so it reflects a tiny batch and is not representative of long-prompt throughput.
• Prompt throughput at 256K reflects large-batch processing of the full 258K-token input.
• Generation speed drops at 256K because the KV cache is roughly twice as large — expected, not a defect.
• At 256K the model followed instructions correctly and stayed coherent across the full context.

Load test: ✅ loads and generates · Vision test: ✅ llama-mtmd-cli + mmproj correctly described a test image · Chat template: ✅ embedded (tokenizer.chat_template)

Usage

llama.cpp CLI (text):

llama-cli -m mimo-v2.6-distill-qwen-9b-mxfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99

llama.cpp CLI (vision):

llama-mtmd-cli -m mimo-v2.6-distill-qwen-9b-mxfp4.gguf \
  --mmproj mmproj-mimo-v2.6-distill-qwen-9b-f16.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 \
  --image your_image.png

llama.cpp server:

llama-server -m mimo-v2.6-distill-qwen-9b-mxfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080

llama-cpp-python: requires a wheel built against a llama.cpp with MXFP4 support — stock PyPI wheels will not load this file. No quantization-aware Python package is published for it yet.

For grounding tasks with images, llama.cpp suggests --image-min-tokens 1024 (upstream issue #16842).

Pipeline

# 1. F16 conversion (--no-mtp is required, see below)
python convert_hf_to_gguf.py <source_dir> --outfile model-f16.gguf --outtype f16 --no-mtp
# 2. vision encoder
python convert_hf_to_gguf.py <source_dir> --outfile mmproj-model-f16.gguf --outtype f16 --mmproj
# 3. quantize
llama-quantize model-f16.gguf model-mxfp4.gguf MXFP4

Model tree / provenance

Qwen/Qwen3.5-9B → XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B → this repo (quantization only).

Quantization does not change the model's weights, knowledge, or training — it only re-encodes them at lower precision. It does not improve the model.

Known limitations

• No MTP head. The source config declares mtp_num_hidden_layers: 1, but the released checkpoint contains no MTP weights, so this GGUF is converted with --no-mtp (block_count: 32). Multi-token prediction is unavailable; this is a property of the released checkpoint, not a conversion loss.
• Thinking is on by default — the upstream template emits [Start thinking] unless the caller disables it.
• This is an SFT research checkpoint, not a polished production model; Xiaomi positions it as a starting point for agentic RL.
• Benchmark numbers are Xiaomi's reported results and were not reproduced during quantization.
• KV cache at 256K is large — on 16 GB cards, Q8 KV leaves limited headroom for long generations.
• Hugging Face may not auto-detect this quantization from the filename (see Quantization notes).