Huihui Qwen3.8-27B Abliterated NVFP4 MTP VL
A compressed-tensors NVFP4 W4A4 release of huihui-ai/Huihui-Qwen3.8-27B-abliterated at revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff. The same-source BF16 vision/video tower (333 tensors) and all 15 BF16 MTP tensors are retained.
Quick start — RTX 5090 / Blackwell
hf download lyf/Qwen3.8-27B-Huihui-Abliterated-NVFP4-MTP-VL --local-dir ./huihui-qwen38-nvfp4
docker run --rm --gpus all --ipc=host --network=host -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass -e VLLM_USE_FLASHINFER_SAMPLER=1 -v "$PWD/huihui-qwen38-nvfp4:/model:ro" vllm/vllm-openai:qwen38-x86_64-cu130 /model --served-model-name qwen38-huihui-nvfp4 --host 0.0.0.0 --port 8000 --max-model-len 4096 --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --max-num-seqs 1 --max-num-batched-tokens 1024 --enable-prefix-caching --trust-remote-code --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
The command above is the verified multimodal/MTP smoke profile. Larger context values require workload-specific VRAM validation.
Quantization and lineage
| Component | Format/source |
|---|---|
| Language-model Linear layers | NVFP4 W4A4, group size 16 |
| Vision/video tower | 333 tensors, BF16, same Huihui checkpoint |
| MTP head | 15 tensors, BF16, same Huihui checkpoint |
lm_head, token embedding, GDN conv1d | BF16 |
| Calibration | CNN/DailyMail 3.0.0, 20 × 8192 tokens |
| Packaging | compressed-tensors nvfp4-pack-quantized |
No official Qwen, Unsloth, Blackfrost, or other behavioral variant weights were grafted into this release.
Validation
Validated on one RTX 5090, 450 W cap, using vllm/vllm-openai:qwen38-x86_64-cu130:
GET /healthand/v1/models: passed- OpenAI-compatible 1024-token text generation: passed
- Native MTP n=3: passed
- MTP draft tokens: 1362; accepted: 572; acceptance: 42.0%
- Mean acceptance length: 2.33
- Per-position acceptance:
0.602 / 0.429 / 0.295 - Client elapsed for 1024 output tokens: 10.73 s
- Runtime VRAM under generation: ~28,944 MiB
The earlier full release workflow also verifies 333 vision tensors and 15 same-source MTP tensors statically. The NInfer derivative in the companion repository carries a separate real-image runtime validation.
Files
model-00001-of-00002.safetensors,model-00002-of-00002.safetensors: compressed checkpointmodel-mtp-extra.safetensors: 15 same-source BF16 MTP tensorsmodel.safetensors.index.json: complete 2687-tensor indexBUILD_MANIFEST.json,VALIDATION_REPORT.json,STATIC_VALIDATION_REPORT.json,recipe.yaml,SHA256SUMS: provenance and reproducibility
Intended use
This is an abliterated behavioral derivative intended for model research and local inference. Users are responsible for downstream use and applicable policies.