Qwen3.5-9BB-MTP-MXFP4-GGUF
About MXFP4
MXFP4 (Microscaling FP4, OCP MXFP4 E2M1) is an open-standard 4-bit format developed by the OCP Microscaling Formats (MX) consortium. Key characteristics:
| Property | MXFP4 | NVFP4 |
|---|---|---|
| Format | E2M1 (1 sign, 2 exponent, 1 mantissa) | E4M3 (1 sign, 4 exponent, 3 mantissa) |
| Block size | 32 elements | 128 elements |
| Hardware | CPU, GPU (all vendors) | NVIDIA Blackwell only |
| Standard | OCP Open Standard | NVIDIA proprietary |
MXFP4 is the dense-model variant (for MoE models, use MXFP4_MOE).
Files
| Filename | Type | Size | Description |
|---|---|---|---|
qwen35-9Bb-mtp-mxfp4.gguf | MXFP4 quantized model | 5.18 GB | Main model weights (MXFP4, 1 MTP head) |
mmproj-qwen35-9Bb-f16.gguf | Multimodal projector | 875 MB | Vision encoder (SigLIP, F16) |
README.md | Documentation | - | This file |
Quantization Details
| Parameter | Value |
|---|---|
| Format | MXFP4 (OCP E2M1) |
| Block size | 32 |
| Bits per weight | 4.72 |
| Hardware target | CPU, AMD GPU, NVIDIA GPU (all), Blackwell |
| VRAM required | ~6.0 GB |
| MTP head | Yes (1 layer, nextn_predict_layers=1) |
Model Description
Qwen3.5-9BB is a multilingual vision-language model with 32 transformer blocks, 262k context length, and 1 MTP (Multi-Token Prediction) head. It supports:
- Text generation (multilingual: EN, ZH, code)
- Vision understanding (image + video)
- Tool calling / function calling
- Thinking mode (reasoning) — enabled by default in this variant
Thinking behavior: This variant has thinking enabled by default. To disable thinking, pass enable_thinking=false in the generation parameters. When disabled, the model outputs only the final answer without reasoning tokens.
Architecture Details
- 32 transformer blocks (hybrid attention + FFN + SSM)
- 1 MTP prediction head (
block_count=33) - 1 SigLIP vision encoder (mmproj)
- 262,144 token context window
- 2560-dim (4B) / 4096-dim (9B) hidden size
Usage
llama.cpp CLI
# Basic text generation (thinking enabled by default)
./llama-cli -m qwen35-9Bb-mtp-mxfp4.gguf \
--mmproj mmproj-qwen35-9Bb-f16.gguf \
-p "What is in this image?" \
--image path/to/image.jpg \
-n 512
# Thinking disabled
./llama-cli -m qwen35-9Bb-mtp-mxfp4.gguf \
--mmproj mmproj-qwen35-9Bb-f16.gguf \
-p "Hello, how are you?" \
-n 256 \
-p "enable_thinking=false"
llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="qwen35-9Bb-mtp-mxfp4.gguf",
mmproj="mmproj-qwen35-9Bb-f16.gguf",
n_ctx=8192,
)
# Thinking enabled by default
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}]
)
# Disable thinking
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"enable_thinking": False}
)
Download from HuggingFace Hub
from huggingface_hub import hf_hub_download
model_path = hf_hub_download(
repo_id="FreedomAISVR/Qwen3.5-9BB-MTP-MXFP4-GGUF",
filename="qwen35-9Bb-mtp-mxfp4.gguf"
)
mmproj_path = hf_hub_download(
repo_id="FreedomAISVR/Qwen3.5-9BB-MTP-MXFP4-GGUF",
filename="mmproj-qwen35-9Bb-f16.gguf"
)
Conversion Pipeline
HF weights (BF16)
→ convert_hf_to_gguf.py (F16, with MTP, no --no-mtp)
→ llama-quantize.exe MXFP4
Hardware
| Component | Value |
|---|---|
| GPU | RTX 5060 Ti (16 GB VRAM) |
| System RAM | 128 GB |
| Quantization time | ~2 min (4B) / ~1.7 min (9B) |
License
Apache 2.0 (same as Qwen3.5-9BB).