Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4-GGUF

IQ4_XS quantization of Qwen3.8-Flash-Next abliterated with Heretic v1.3.0+custom (fork), with NGQ4-compressed n-gram prediction table. Built and benchmarked on AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, 128 GB unified).

Files

Quantization

Quantized from the Heretic2 BF16 safetensors → Q8_0 intermediate → IQ4_XS using the llama-flash-next fork of llama.cpp (build 10656, commit 035e22731). The NGQ4 step compresses the 54 GB per-layer n-gram prediction table to ~27 GB by lowering precision on the indexer's token embedding, reducing the model from 116 GB to 92 GB with no measurable accuracy impact through 32K context.

Byte-verified against the Heretic2 BF16 checkpoint.

Tensor types

How to run

Requires the llama-flash-next fork (qwen4exp branch) — standard llama.cpp cannot load the qwen4exp architecture.

# Text only
./llama-server \
  -m Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4.gguf \
  --alias qwen38-flash-next-heretic2 \
  -ngl 99 -fa on -c 65536 -np 1 --jinja \
  --host 127.0.0.1 --port 8096 \
  -dev Vulkan0

# With vision
./llama-server \
  -m Qwen3.8-Flash-Next-Heretic2-IQ4XS-NGQ4.gguf \
  --mmproj mmproj-Heretic2-BF16.gguf \
  --alias qwen38-flash-next-heretic2 \
  -ngl 99 -fa on -c 65536 -np 1 --jinja \
  --host 127.0.0.1 --port 8096 \
  -dev Vulkan0

Thinking mode is optional — disable it with "chat_template_kwargs": {"enable_thinking": false} in API requests to use the full token budget for output.

Benchmarks — HumanEval+ (EvalPlus)

164 problems, enable_thinking: false, temperature: 0.0, max_tokens: 1024.

This model offers the best accuracy-to-speed ratio in the IQ4_XS tier: 84.1% base at 4.16s per problem. Only the Q6_K variants score higher, but they require 129 GB (won't coexist with other workloads on 128 GB) and take 50–170% longer per response.

Origin story: Why this model exists

This GGUF was originally built as part of an apples-to-apples comparison between llama-server and Halogen Flash Server on AMD Strix Halo. Halogen's prefill throughput is dramatically faster — 2–4x at short contexts, approaching 4x at 16K+ tokens — so the question was whether it could replace llama-server as the default inference engine.

The answer turned out to be no, because of a critical limitation in Halogen's BYO (bring-your-own) GGUF workflow:

The overlay problem

Halogen's official checkpoint ships with a quality overlay sidecar (qwen38-flash-next-w4b.overlay.hgn, 2.4 GiB, 741 activation-aware re-quantized tensors). The official model with overlay scores 82.3% / 78.0% on HumanEval+ — competitive with llama-server. But the overlay is built through an internal activation-aware quantization pipeline that isn't publicly available. There's no halogen-quantize --build-overlay command.

When you bring your own GGUF — any fine-tune, any abliteration, any custom variant — you run without the overlay. The result: 65.2% / 61.0%, a 17-point collapse from the official model, using the exact same Heretic2 weights that score 84.1% on llama-server.

The thinking mode problem

Halogen forces thinking mode on Qwen3.8-Flash-Next with no way to disable it. Every request burns ~1,049 hidden reasoning tokens before producing visible output. With max_tokens=2048, that leaves ~999 tokens for code. The official model with overlay compensates by producing better code in fewer tokens. Without the overlay, the BYO model truncates — 68% of HumanEval+ failures were SyntaxErrors from incomplete code that the model reported as finish_reason=stop.

llama-server doesn't have this problem: thinking mode is optional via enable_thinking: false, and the full token budget goes to output.

The outcome

This Heretic2 IQ4_XS GGUF — originally built just for comparison — turned out to be a better model for production deployment than either the Halogen BYO setup (fast but 65% accuracy) or the Halogen official (82% accuracy but 3x slower per problem due to forced thinking). It runs on llama-server with full control over thinking mode, optional vision, and 84.1% HumanEval+ accuracy at 4.16s per problem.

For the full technical assessment, see the companion write-up: Halogen on Strix Halo: Fast but Not Ready

Abliteration details

The Heretic2 abliteration was performed using Heretic v1.3.0+custom (fork by timrohrbaugh) on the base Qwen/Qwen3.8-Flash-Next weights. Direction-removal method with per-layer direction index.

The abliteration has no measurable impact on code generation accuracy — Heretic2 IQ4_XS scores 84.1% vs base IQ4_XS at 82.3%, and the plus tests are within noise (79.3% vs 78.0%).

Hardware tested

• CPU: AMD Ryzen AI Max+ 395
• GPU: AMD Radeon 8060S (gfx1151, RDNA 4 integrated)
• Memory: 128 GB unified (DDR5)
• Backend: Vulkan (ROCm also works but is slower for this architecture)
• OS: Ubuntu

Throughput on Strix Halo (llama-server, Vulkan)

Prefill (IQ4_XS standard, same architecture)

Decode

~24 t/s serial (memory-bandwidth limited on unified memory).

License

Qwen Community License 1.0 (inherited from the base model).

Credits

• Base model: Qwen/Qwen3.8-Flash-Next by Alibaba Qwen team
• Abliteration: Heretic by p-e-w, custom fork by timrohrbaugh
• Quantization fork: llama-flash-next (qwen4exp branch) by nicoboss
• Quantization & benchmarking: cygnal