Qwen3.8-Flash-Next-Heretic2-IQ4XS-Halogen

A Halogen Flash Server BYO GGUF quantization of Qwen3.8-Flash-Next, abliterated via Heretic v1.3.0 (the "Heretic2" build). This GGUF is specifically built to run on Halogen Flash Server (peonist-ai) with custom gfx1151 kernels on AMD Strix Halo hardware. It uses a carefully tuned mix of tensor types that Halogen accepts for its BYO GGUF mode.

Now with vision support — pair with the official vision tower from peonist-ai/halogen-qwen3.8-flash-next for multimodal (image) input.

Key Specs

Tensor Type Map

Vision / Multimodal Support

Qwen3.8-Flash-Next is natively multimodal (text + image). To enable vision with this BYO GGUF:

1. Download qwen38-flash-next-vision.hgn (0.84 GiB) from peonist-ai/halogen-qwen3.8-flash-next
2. Place it in your models directory alongside the GGUF
3. Add -e HALOGEN_VISION_TOWER=/models/qwen38-flash-next-vision.hgn to your Docker run command

The vision tower is a separate encoder that runs independently of the language model weights — it works the same whether you're running the native HGN checkpoint or a BYO GGUF.

Vision Performance (Strix Halo, warm tower)

• Images sent as base64-encoded PNG via OpenAI-compatible API
• HALOGEN_VISION_MAX_PIXELS controls auto-scaling (default: 2560x1440)
• Text at 12pt+ is read accurately at all supported resolutions
• Video is not currently supported

Benchmarks: Halogen vs llama-server (Strix Halo)

Hardware: AMD Ryzen AI Max+ 395, Radeon 8060S (gfx1151), ~117 GB unified memory

Prefill Throughput (tok/s)

Methodology: unique random prompts per run (no KV cache hits), max_tokens=1 to isolate prefill, 3 runs per context length, mean reported.

llama-server uses llama.cpp's qwen4exp branch (build 10656) with a standard IQ4_XS GGUF (116 GB). Halogen uses its BYO GGUF (93 GB) with custom tensor type layout. Both are IQ4_XS-level quantization but different GGUF formats. The Q8_0 source GGUF (176 GB) does not fit in memory on this 128 GB system.

Decode Throughput (tok/s)

MTP (multi-token prediction) is a decode-phase optimization only — the Halogen API reports draft_n=0 during prefill.

Key Takeaways

• Prefill gap widens with context: Halogen is 1.45x faster at 512 tokens, growing to 3.68x at 16K — llama-server's throughput decreases at longer contexts while Halogen's increases
• Decode is close in serial: Both ~24 t/s (memory-bandwidth limited on unified memory)
• MTP decode wins: ~32 t/s vs ~24 t/s serial, ~33% boost from speculative drafting
• Half the VRAM: 93 GB vs 176 GB Q8_0, freeing memory for longer contexts

How to Run

Requirements

• Halogen Flash Server v0.11.4+
• AMD GPU with ROCm (tested on gfx1151 / Radeon 8060S)
• ~117 GB system RAM (unified memory architecture)
• MTP head file: qwen38-flash-next-mtp.hgn (from peonist-ai, 1.5 GB)
• Vision tower (optional): qwen38-flash-next-vision.hgn (from peonist-ai, 0.84 GB)

Docker

docker run -d \
    --name halogen-server \
    --restart unless-stopped \
    --network=host \
    --device=/dev/kfd \
    --device=/dev/dri \
    -v /path/to/models:/models:rw \
    -v /path/to/tokenizer:/tokenizer:ro \
    -e HALOGEN_ARCH=gfx1151 \
    -e HALOGEN_CHECKPOINT=/models/heretic2.gguf \
    -e HALOGEN_TOKENIZER=/tokenizer \
    -e HALOGEN_MTP_HEAD=/models/qwen38-flash-next-mtp.hgn \
    -e HALOGEN_VISION_TOWER=/models/qwen38-flash-next-vision.hgn \
    -e HALOGEN_KV_SLOTS=4 \
    -e HALOGEN_CTX=262144 \
    -e HALOGEN_MAX_TOK=32768 \
    -e HALOGEN_PREFILL_CHUNK=32768 \
    -e HALOGEN_GGUF_CACHE=1 \
    -e HALOGEN_API_PORT=8096 \
    ghcr.io/peonist-ai/halogen-flash-server:0.11.4 \
    all

Omit HALOGEN_VISION_TOWER for text-only mode (byte-identical text behavior).

Build Process

Built from the Heretic2 BF16 weights (Qwen3.8-Flash-Next abliterated via Heretic v1.3.0 direction-removal):

1. BF16 to Q8_0: Standard llama.cpp quantization (176 GB)
2. Q8_0 to IQ4_XS-Halogen-v8: Re-quantized with llama-quantize --allow-requantize and per-tensor type overrides

The critical discovery during this build: per_layer_token_embd.weight MUST be IQ4_NL for Halogen. This 26.8 GiB tensor is read in-place via mmap by Halogen's k_embed_gather kernel — it is NOT repacked at startup. Using Q8_0 (50.7 GiB, different block structure) causes a GPU page fault and crash. The unsloth official UD-IQ4_XS GGUF confirms this tensor is type_id=20 (IQ4_NL).

Quantize Command

llama-quantize --allow-requantize \
  --token-embedding-type bf16 \
  --tensor-type "per_layer_token_embd=iq4_nl" \
  --tensor-type "^output\.weight=q6_k" \
  --tensor-type "ffn_down_exps=iq4_xs" \
  --tensor-type "ffn_up_exps=iq4_xs" \
  --tensor-type "ffn_gate_exps=iq4_xs" \
  --tensor-type "ffn_down_shexp=iq4_xs" \
  --tensor-type "ffn_up_shexp=iq4_xs" \
  --tensor-type "ffn_gate_shexp=iq4_xs" \
  --tensor-type "hc_ffn_down=iq4_xs" \
  --tensor-type "hc_ffn_up=iq4_xs" \
  --tensor-type "hc_attn_down=iq4_xs" \
  --tensor-type "hc_attn_up=iq4_xs" \
  --tensor-type "hc_attn_inject=iq4_xs" \
  --tensor-type "hc_ffn_inject=iq4_xs" \
  --tensor-type "ple_conv1d=f32" \
  --tensor-type "indexer.*proj=bf16" \
  source_q8.gguf output.gguf q8_0

Note: Requires a patched llama-quant.cpp with lines 330-331 commented out (hardcoded indexer projection exclusions).

Halogen-Accepted Tensor Types (BYO GGUF)

Rejected: K-quants (Q4_K, Q5_K, Q5_1, Q4_1), IQ2/IQ1 families, F16

Credits

• Base model: Qwen/Qwen3.8-Flash-Next by Alibaba Qwen team
• Abliteration: Heretic v1.3.0 direction-removal method
• Inference engine: Halogen Flash Server by peonist-ai
• Quantization tooling: llama.cpp (patched)
• Hardware: AMD Strix Halo (Ryzen AI Max+ 395)