Qwen3.8-27B-MLX-bf16-mtp
Full-precision bf16 MLX conversion of Qwen/Qwen3.8-27B with the native MTP (multi-token prediction) head preserved, built for the oMLX inference engine on Apple Silicon.
- Complete VLM: vision tower included (image + video understanding).
- MTP-preserved: all 15
mtp.*tensors kept (mtp_num_hidden_layers: 1), enabling Lightning-MTP speculative decoding in oMLX (mtp_enabled: true). - bf16 everywhere (~52 GB weights): no quantization; the Gated DeltaNet recurrent state self-promotes to fp32 internally at runtime, so bf16 storage is lossless in practice.
- Norm convention: weights are in MLX's shifted (+1) RMSNorm convention, including all MTP-head norms — safe for oMLX's per-key convention detection (no double shift).
Conversion details
- Converter:
mlx_vlm.convert(mlx-vlm 0.6.3, mlx 0.32.0) run with oMLX 0.5.7'smlx_vlm_mtppatches applied (apply_mlx_vlm_mtp_runtime_patch()+apply_mlx_vlm_mtp_patch()before load), which is what preserves the MTP head — stockmlx_vlm.convertstripsmtp.*. dtype="bfloat16",quantize=False.- Source revision:
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
Measured on an M5 Max (128 GB), oMLX 0.5.7
| Config | Decode speed | Notes |
|---|---|---|
| MTP off | ~8.9 tok/s | plain bf16 decode |
| MTP on (depth 3) | ~20.3 tok/s (2.3×) | 71–89% draft acceptance, 2.5–3.2 tok/backbone-cycle |
31K-token prompt prefills in ~40 s (GDN chunked Metal kernel path); needle retrieval and image grounding verified.
Usage with oMLX
Place under ~/.omlx/models/<org>/Qwen3.8-27B-MLX-bf16-mtp (or download via the oMLX admin dashboard), then enable MTP for the model:
"Qwen3.8-27B-MLX-bf16-mtp": { "mtp_enabled": true, "max_context_window": 262144 }
oMLX routes it to the VLM engine automatically and selects the Lightning MTP speculative backend. Sampling defaults from generation_config.json: temperature 1.0, top_p 0.95, top_k 20.
If you want a standalone drafter for oMLX's external vlm_mtp_enabled path instead, you can extract one from this checkpoint's source with python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split (emits model_type: qwen3_5_mtp, block_size: 3, ~830 MB). The in-checkpoint native path above is faster and is the recommended default.
The model also works with stock mlx-vlm ≥ 0.6.3 as a plain bf16 VLM (the mtp.* tensors are ignored by loaders that don't implement MTP).