About this conversion
This release uses a 2-bit affine base and assigns more precision to sensitive modules. The final allocation includes 3, 4, 5, 6, and 8-bit modules. The native Qwen4Exp MTP block uses the same mixed-precision approach and remains part of the checkpoint.
| Item | Value |
|---|---|
| Repository | Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP |
| Base model | Qwen/Qwen3.8-Flash-Next |
| Source weights | Official BF16 checkpoint |
| Format | MLX safetensors |
| Quantisation | Sensitivity-aware oQ2 mixed precision |
| Base allocation | 2-bit affine, group size 32 |
| Module overrides | 297 at 3-bit, 60 at 4-bit, 36 at 5-bit, 129 at 6-bit, 224 at 8-bit |
| Override group sizes | 32, 64, or 128 according to module |
| Native MTP | Included, one Qwen4Exp draft block |
| Indexed tensors | 3,747 total, including 76 MTP entries |
| Weight shards | 15 |
| Weight size | 70.782 GB / 65.921 GiB |
| Configured context | 262,144 tokens |
| Architecture | qwen4_exp vision-language sparse MoE |
The upstream tokenizer, chat template, image and video processor configuration, generation configuration, licence, and native MTP configuration are included.
[!IMPORTANT] Use a runtime with explicit
qwen4_expand native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP entries during strict loading.
Download and use
python -m pip install --upgrade huggingface_hub
hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP \
--local-dir ./Qwen3.8-Flash-Next-MLX-oQ2-MTP
Add the downloaded directory to a compatible oMLX model directory and refresh the model registry. Native MTP is optional. Keep it disabled by default for this release because the measured depth-three path was slightly slower than baseline.
Apple M3 Studio performance
The validation used three measured 512-token runs per mode. The table reports median decode throughput.
| Runtime mode | Runs | Output per run | Median generation speed |
|---|---|---|---|
| Native MTP disabled | 3 | 512 tokens | 26.1398 tokens/s |
| Native MTP enabled, depth 3 | 3 | 512 tokens | 25.9290 tokens/s |
Native MTP changed median throughput by -0.81% in this test. The telemetry sample accepted 33 of 57 draft proposals, an acceptance rate of 57.89%. Each of the three paired MTP-off and MTP-on 512-token runs produced matching output hashes.
Instruction following, factual recall, arithmetic, and coherent long-generation gates passed in both modes. MTP worked correctly, but it did not improve throughput on this checkpoint and test. The recommended default is therefore MTP disabled.
The benchmark covers text generation. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions.
Architecture
Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.
| Architecture detail | Upstream value |
|---|---|
| Language-model parameters | 125B total / 6B active |
| N-gram embedding | 51B parameters, 20,000,000 entries |
| Native MTP | 4B parameters, one draft layer |
| Hidden size | 2,560 |
| Layers | 48 |
| Routed / active experts | 512 / 10, plus 1 shared expert |
| Native context | 262,144 tokens, extensible upstream to 1,000,000 |
For upstream evaluations, intended use, safety guidance, and the full architecture discussion, see the original model card.
Conversion and validation
- The converter read the official BF16 checkpoint directly.
- The recipe used a 2-bit affine base with 746 mixed-precision module overrides.
- Full checkpoint validation covered all 3,747 indexed tensors, all 15 shards, and all 76 native-MTP tensor entries.
- Three deterministic 512-token runs passed in each MTP mode with exact paired output parity.
- Instruction, factual, arithmetic, and coherence checks passed in both modes.
- Native-MTP telemetry captured the draft and acceptance counters.
This is a community conversion, not an official Qwen release.
Limitations
- oQ2 is an extreme-compression format. Evaluate instruction following, reasoning, factual accuracy, and visual understanding on the intended workload before deployment.
- Native MTP was 0.81% slower in the measured test. Acceptance alone does not guarantee a throughput gain.
- The configured 262,144-token context does not guarantee that every Apple-silicon system can run that length within available unified memory.
- This is an MLX checkpoint for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
- Upstream model limitations and safety considerations still apply.
Licence and attribution
The upstream model is released under the Qwen Community License 1.0. The required licence text is included in this repository and should be reviewed before use or redistribution.
Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by Vontra.
Choose for your Mac
64GB Macs · 128GB Macs · 256GB Macs
No measured memory tier is assigned here. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.
Runtime and evidence
The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.
Quick start and demo prompt
hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP --local-dir ./models/Qwen3.8-Flash-Next-MLX-oQ2-MTP
Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.
Try this in a new chat with a 128-token output limit:
Explain why the sky looks blue in three short sentences.
This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.