Huihui-Qwen3.8-27B Abliterated NVFP4-MTP GGUF

NVFP4 quantized Huihui-Qwen3.8-27B (abliterated / uncensored fine-tune of Qwen3.8-27B) with MTP (multi-token prediction) draft layers, packaged as a single GGUF for llama.cpp speculative decoding on Blackwell GPUs (RTX 5090 / 5080).

What this is

• Quantization: NVFP4 (weights + KV cache), converted to GGUF via llama.cpp convert_hf_to_gguf.py --outtype auto
• MTP draft layers included — enables --spec-type draft-mtp --spec-draft-n-max 2 in llama.cpp for speculative decoding
• Multimodal: pairs with the mmproj-F16.gguf vision projector (from unsloth/Qwen3.8-27B-GGUF)
• Single file: huihui-qwen3.8-27b-abliterated-nvfp4-mtp.gguf (~19 GB)

Attribution

• Source weights: sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4 (Apache-2.0) — this repo is a mechanical GGUF conversion of that repo's safetensors; all credit for the abliterated fine-tune and NVFP4 quantization goes to sakamakismile
• Base model: Qwen/Qwen3.8-27B

Conversion methodology

Converted on an RTX 5090 (CachyOS, user-space, no Docker). The conversion script lives in the felippeburk/rtx-5090-cachyos-testing repo:

# Builds llama.cpp with NVFP4/MTP support, downloads source safetensors,
# runs the conversion, and verifies the result. Everything installs to ~/.local.
git clone https://gitlab.com/felippeburk/rtx-5090-cachyos-testing
cd rtx-5090-cachyos-testing
bash llama/setup-mtp.sh --model huihui

The conversion step is: llama.cpp/convert_hf_to_gguf.py <source-safetensors-dir> --outfile huihui-qwen3.8-27b-abliterated-nvfp4-mtp.gguf --outtype auto, run with torch + transformers in an isolated venv. Reproducible from the source safetensors at any time.

Recommended server args (llama.cpp, 192K context)

-m huihui-qwen3.8-27b-abliterated-nvfp4-mtp.gguf \
--mmproj mmproj-F16.gguf \
-fitt 8192 -c 196608 -n 65536 -fa on -ngl 99 -np 1 -t 16 -tb 16 \
-ctk q8_0 -ctv q8_0 -ctkd q4_1 -ctvd q4_1 -ctxcp 16 -cram 6144 \
--cache-idle-slots --no-warmup \
--spec-type draft-mtp --spec-draft-n-max 2 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
--jinja --chat-template-file config/chat_template.jinja \
--reasoning on --reasoning-format deepseek --reasoning-budget 4096 \
--reasoning-budget-message "I have reached my reasoning budget. I will now provide my final answer or take the most appropriate action based on my analysis so far." \
--chat-template-kwargs '{"reasoning_effort":"medium"}'

Important usage notes

• Use reasoning_effort medium. This abliterated fine-tune was validated at medium — at xhigh the think phase can run away past the token budget and never terminate. --chat-template-kwargs forces medium above; a patched chat template may default to xhigh.
• Thinking mode requires a generous output budget. This model uses the Qwen3.8 thinking format, so reasoning shares the max_tokens budget with the answer. Set the client's max_tokens to at least 65536 (-n 65536 above) and give the server a --reasoning-budget (here 4096) so the thinking phase can't eat the entire output. With a too-small max_tokens, long prompts can produce reasoning but an empty final answer.
• Pin a reasoning-effort budget message. --reasoning-budget-message tells the model to wrap up and answer once the budget is hit; without it, long tool-calling sessions can stall in thinking.
• Use thinking-mode sampling. Temperature 1.0, presence_penalty 0.0 (the 0.7 / 1.5 values in some older configs are for instruct mode and hurt thinking-mode output).
• reasoning_effort only accepts xhigh / medium / low — there is no high level.
• Vision is available via --mmproj mmproj-F16.gguf (downloaded from unsloth/Qwen3.8-27B-GGUF).

Full config lives in config/models.yaml of that GitLab repo.

License

Apache-2.0 (inherited from source weights and base model)