Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

The balanced Qwen3.8-27B quant for AMD Strix Halo (Ryzen AI Max+ 395 / gfx1151). Measured, not marketed: one 15 GB file on the iGPU, the full 262,144-token window integrity-checked, three serving profiles — reading 101–410 t/s and writing 14.8–49.8 t/s across the window (depending on how full it is, what you ask for, and the profile).

Reading = processing your prompt (prefill). Writing = producing the answer (decode). t/s = tokens per second.

Brand: PaoAI · community quant, not affiliated with Alibaba or the Qwen team.

Updated 2026-09-29 — new engine version + three serving profiles, weights unchanged. The latest version runs on paoai-strix-engine strix-v4 (e9b93d56a) with three ready-made settings: Chat (default), Code (with a small helper model) and Long (for long documents and long code). Same file, same sha256. The speed tables and the quality bench were measured on the published engine; the coding exam and the 256K needle ran on the same code a day earlier (noted where they appear). The previous version's numbers are kept only in the "against the previous version" table. See Update log.

What is this? (30 seconds)

Qwen3.8-27B is a hybrid: 48 cheap linear-attention layers and only 16 full-attention layers that actually re-read your context. We compressed it part by part: the finer 4.5-bit FP4 layout where it matters most (attention K/V and the linear-attention qkv), the faster 4.25-bit layout for the bulk, 6-bit token embeddings, every norm at full precision — guided by an importance matrix. One 15 GB file, no offload flags, speculative decoding built in, checked from 8K to the full 256K window.

The recipe (what's inside — read from the file)

Total: 15.01 GB, 27.3 B weights, 4.40 bits per weight on average. (Measured from the published file's tensor table. Our earlier card listed the draft head as Q8_0 and the whole of full attention as one format; the file has always been as above — same sha256.)

Which profile? (the same file, three ways to serve it)

All three give the same answers the model would give without any speed-up, within rounding (checked below). Pick by what you do most.

Coding exam — "chain test" (3 runs per profile, fresh server each — latest version, 2026-09-28)

A fixed coding exam graded by running the code, not by opinion: the model reads deceptive code and must trace it, implements a novel state machine, and writes a thread-safe ring buffer that survives a 5,000-operation stress test. Three identical runs; we report medians with the spread (identical runs vary — single scores are unreliable).

Code and Long scored 100 on every task in all three runs. Chat's run totals were 100 · 73 · 53: in two of them the ring-buffer code crashed on Condition.notify_one(), a method Python does not have (the hunt miss in the third run was a trace step). We checked it with more runs, rules fixed before running: over 10 Chat runs the ring buffer failed 3 times (2× notify_one, 1× notify without holding the lock); with this card's previous settings it failed 0 of 6 (those runs slipped on other tasks instead). The original full-precision model (BF16, no quantization, a cloud server, 5 runs) wrote the same notify_one crash 1 time and failed the ring buffer 2 times in all — so it is the base model's habit, not something the quant or the Chat settings add. The gap between 3 of 10 and 0 of 6 is within chance (Fisher p ≈ 0.25).

Writing speed per phase (tokens/second, range over the 3 runs)

Short prompts (a few hundred tokens). Exam settings: 64K window, thinking budget 1,500 tokens, each profile on its own settings. Run 2026-09-28 on our test build of the exact engine code that was then published as strix-v4 (same source tree, checked).

Does quantization hurt? (the file itself, unchanged by this update — measured 2026-09-17)

Plain words: the chain test above can only tell you if the model still passes. This test asks a different question: did compressing the weights change what the model actually predicts? We ran the industry-standard check — llama.cpp's own llama-perplexity tool — comparing this 15 GB FP4 file against a full-precision F16 copy of the same model (built by us from Qwen's official BF16 release, sha-verified shards), reading the same ~295,000 tokens of standard Wikipedia text through both at 8K windows.

Plain read: on standard text, the FP4 cut is statistically indistinguishable from the full-precision model — no measurable quality loss. The small difference that exists is concentrated in a tail of rare positions (~1 % of tokens shift noticeably), and the chain test above shows those shifts didn't cost task success. Not lossless in the mathematical sense — lossy in a way we cannot measure on standard text.

Reproducible: llama.cpp llama-perplexity, engine commit 11bfe8a, wikitext-2-raw, -c 8192, 36 chunks (~295K tokens). Reference F16 converted by us from Qwen's official BF16 release (sha-verified shards). Full data in our results file.

Do the speed-ups change the answers? (latest version, 2026-09-28)

Speculative decoding should never change what the model writes: the big model checks every guessed token. We checked it anyway, on 22 fixed short and code questions at temperature 0, against the same engine with no speed-up:

Deep-context sweep — what happens when the window fills (8K → 256K)

We filled the window in five steps and measured reading and writing at each — a fresh server and a fresh prompt at every point (no prompt-cache reuse), three kinds of answer per depth, one pass, for each profile.

Chat profile

Code profile

Long profile

The three answer kinds: short answer = "What is 17 times 23 plus N? Answer with the number only." (up to 300 tokens, thinking on — the method of every earlier table on this card; the answer is checked); code = "write a complete Python module … with tests" and long writing = "write a long, detailed explanation … 900+ words", each exactly 1,000 tokens, thinking off. Reading = the short-answer prompt. Measured 2026-09-29 on one Strix Halo, latest version (binary built from 8ddb7c4d3; the branch tip e9b93d56a only adds README fixes), window -c 73,728 up to 64K, 139,264 at 128K, 262,144 at 256K (the window size does not change the speed at a given fill), prompt cache off, -ub 256 at 256K. Achieved fills: 8,238 / 32,813 / 65,594 / 131,127 / 260,056 tokens. Speeds are the server's own timings.

Against the previous version (engine 4905ff417, the settings on this card before, short-answer method — the only kind the old table has):

Previous-version numbers: this card's 2026-09-20 table (same short-answer question, one Strix Halo, previous engine and settings). A short answer is a few hundred tokens, so its writing speed moves more from run to run than the 1,000-token rows.

Full window — 262,144 tokens (latest version)

The model's own metadata declares a 262,144-token window (qwen35.context_length). A needle test passed on every profile: one sentence ("Note: The vault code is 7291-ALPHA.") hidden in the middle of the filled window → asked for the code: Chat 7291-ALPHA at 260,018 tokens; Code 7291-ALPHA at 260,018 tokens; Long 7291-ALPHA at 260,018 tokens. (Run 2026-09-28 on our test build of the same engine code, before the branch was published; every other number in this section is from the published build.)

Serve it with -c 262144 for the full window, and add -ub 256 for prompts past ~200K — the default batch can lose the Vulkan device there (vk::DeviceLostError, reproduced at ~213K); with -ub 256 all three profiles were clean at 260K.

Quality bench (PaoAI Quality Bench v1.3 — 55 tasks, 7 subjects, Chat profile)

Overall 83.2 (code counts ×2) · one run per task · temperature 0 · one request at a time · 87.1 min total · latest version.

Short prompt: most tasks send 60–270 tokens, so their reading speed is mostly start-up time; real reading speed = the long-context row and the sweep above.

Out of budget: this model thinks at length. On 6 of 55 tasks it used its whole token budget (code 2, math 2, realworld 2) and scored 0 — still thinking, or its answer was cut off. That is how the model behaves at these budgets (math 2,048 · real world 8,192 · code 12,288–16,384 tokens), not a crash. Same budgets for every model we test, no retries.

Settings: Chat profile, thinking budget 12,000 (the bench standard; the rest of this card uses 1,500), -c 262144, latest version.

Facts

Requirements — which llama.cpp build (latest version)

This model uses two things stock llama.cpp does not have: the ROCmFP4 tensor types (types 100 and 101) and, for the Chat profile, the draft word list switch. The qwen35 architecture itself is in upstream llama.cpp.

• Latest version: guevae2/paoai-strix-engine, branch strix-v4 @ e9b93d56a (e9b93d56a30ea24001ac4380209c6daca6df9a66). The speed numbers were measured on a build of 8ddb7c4d3; the later commits change only the README. It is our strix-main (afd1088e8, the engine behind the previous version, a fork of LaurentZuijdwijk/llama.cpp) plus three changes: LLAMA_FA_VERIFY_PACK (faster check of the guesses — on by default in this build, =0 turns it off), LLAMA_MTP_DRAFT_VOCAB (the word list — off unless set) and GGML_VK_FA_STAGING (experimental, off unless set, no profile uses it). LLAMA_KV_ROW_PAD was already in strix-main.
• Previous version: the same repo, branch strix-main @ 4905ff417 — the build this card's earlier numbers were measured on. This file runs on it unchanged (the Chat profile, minus the word list and the faster check).

How to run (latest version, measured)

What you need: an AMD Strix Halo PC (Ryzen AI Max+ 395, Radeon 8060S), running Linux (this engine has no Windows build), and about 20 GB of free disk (model 15.0 GB + helper 2.1 GB + engine build 0.6 GB). Memory: at the full 262,144 window the server used about 28 GiB (Chat), 30 GiB (Code) and 40 GiB (Long) — measured on a 128 GB machine; we have not tested a 64 GB one.

1. Install the tools (once).

Fedora:

sudo dnf install -y ca-certificates cmake curl gcc gcc-c++ git glslc python3 spirv-headers-devel vulkan-loader-devel \
  mesa-vulkan-drivers

Ubuntu / Debian:

sudo apt update && sudo apt install -y build-essential ca-certificates cmake curl git glslc libvulkan-dev spirv-headers \
  mesa-vulkan-drivers python3-venv

Then, on both:

curl -LsSf https://hf.co/cli/install.sh | bash     # the hf download tool
export PATH="$HOME/.local/bin:$PATH"               # so this terminal finds hf (new terminals do it by themselves)

2. Build the engine (about 10 minutes, no sudo):

git clone -b strix-v4 https://github.com/guevae2/paoai-strix-engine
cd paoai-strix-engine
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
cmake --build build --config Release -j --target llama-server
cd ..

It ends with [100%] Built target llama-server.

3. Get the files (anonymous, no login needed). The helper is only needed for the Code and Long profiles:

hf download PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf --local-dir .
hf download z-lab/Qwen3.8-27B-DFlash2-GGUF Qwen3.8-27B-DFlash2-Q8_0.gguf --local-dir .

4. Run it (from the folder with the files; pick ONE profile):

Chat (the default — chat, writing, mixed use):

LLAMA_FA_VERIFY_PACK=1 LLAMA_MTP_DRAFT_VOCAB=paoai-strix-engine/paoai/draft-vocab-v2-65536.bin \
paoai-strix-engine/build/bin/llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  -ngl 99 -fa on -c 262144 --jinja --reasoning-budget 1500 \
  -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 4 \
  --host 0.0.0.0 --port 8080

Code (coding agents, up to ~64K context):

LLAMA_FA_VERIFY_PACK=1 \
paoai-strix-engine/build/bin/llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  -ngl 99 -fa on -c 262144 --jinja --reasoning-budget 1500 \
  -ctk q8_0 -ctv q8_0 -md Qwen3.8-27B-DFlash2-Q8_0.gguf -ngld 99 \
  --spec-type draft-dflash --spec-draft-p-min 0.10 --spec-draft-adaptive --spec-draft-n-min 3 --spec-draft-n-max 7 \
  --host 0.0.0.0 --port 8080

Long (long documents and long code, 64K+):

LLAMA_FA_VERIFY_PACK=1 LLAMA_KV_ROW_PAD=256 \
paoai-strix-engine/build/bin/llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  -ngl 99 -fa on -c 262144 --jinja --reasoning-budget 1500 \
  -ctk f16 -ctv f16 -md Qwen3.8-27B-DFlash2-Q8_0.gguf -ngld 99 \
  --spec-type draft-dflash --spec-draft-p-min 0.10 --spec-draft-adaptive --spec-draft-n-min 3 --spec-draft-n-max 7 \
  --host 0.0.0.0 --port 8080

After about 20 seconds the log says listening on http://0.0.0.0:8080. Then open http://localhost:8080 in a browser for the built-in chat page, or point any OpenAI-compatible app at http://localhost:8080/v1. The server also answers other computers on your network (that is what --host 0.0.0.0 does); use --host 127.0.0.1 to keep it to this PC only.

If something goes wrong

Steps 1–4 were run exactly as printed, on 2026-09-29, on a clean Fedora 44 and a clean Ubuntu 24.04 (fresh systems, a normal user, the GPU passed in): both built the engine, downloaded both files with no login (checksums matched), served all three profiles, and answered correctly.

Optional: check the files (the sums must match):

sha256sum Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf   # 3727594804afef8140165e36c0919c70a98b43751a3153d43dd10385c1f100fd
sha256sum Qwen3.8-27B-DFlash2-Q8_0.gguf                   # c18e800daedc59ca68fd13b6a856d795746af6d399a9279ac6a277d1d422f87e

Known limits (honest)

• Depth costs speed: reading and writing slow down as the window fills (tables above). The 16 full-attention layers re-read the history for every token.
• The Code helper is for code: on long free text at 8K–32K it writes slower than Chat (8K: 28.6 vs 36.9 t/s). Pick Chat for chat and writing, Code for code, Long for long inputs.
• The model sometimes writes notify_one() (a C++/Java name) in Python threading code — see the coding exam. The full-precision model does it too; check threaded Python it writes.
• Vulkan only: every number here is from the gfx1151 Vulkan backend on a Strix Halo iGPU. CPU builds and other GPUs are not comparable.
• Long thinking: a bigger --reasoning-budget means noticeably longer waits before the answer starts.
• Not adopted, and why: adaptive drafting on the built-in draft head (measured slower); wide batches -ub 2048 (no gain, and it lost the GPU device at depth).

Serving features explained (plain words)

Every switch in the run commands above, from this build's own llama-server --help and source:

Update log

Earlier engine numbers are not repeated here — they were measured on a different build; the tables above are the current measurement. If you reproduce a number, quote the engine commit with it.

Fleet recipes

Serving configs and our benchmark battery: paoai-strix-recipes (GitHub).

Credits & license

• Qwen team (Alibaba) — base model Qwen/Qwen3.8-27B (Apache-2.0)
• ggml-org — llama.cpp: the engine, the quant formats, the Vulkan backend, DFlash and adaptive drafting
• Laurent Zuijdwijk — LaurentZuijdwijk/llama.cpp (master @ 11bfe8a): the Strix-Halo-tuned Vulkan fork paoai-strix-engine is built on, including the adaptive draft length the Code and Long profiles use (written with his coding agent agention-marshall)
• nathanw — Nathanw1014/llama.cpp: ROCmFP4 types, qwen3.5/MTP work, and the coopmat GEMM rewrite (e68dd31db) carried in this engine
• kingjones30 / charlie12345 — kingjones30/ROCmFPX · charlie12345/ROCmFPX: ROCmFPX format & recipes
• vcruz305 — vcruz305/llama.cpp: converter branch work (qwen3_5 support path)
• z-lab — z-lab/Qwen3.8-27B-DFlash2-GGUF (Apache-2.0): the DFlash2 helper model the Code and Long profiles use
• FR-Spec authors — FR-Spec: frequency-ranked speculative sampling, and Shreesh-Sree — llama.cpp PR #29143 (draft-vocabulary trim for qwen35 MTP): the idea behind the Chat profile's draft word list
• Unsloth — huggingface.co/unsloth: imatrix methodology reference
• deepu105 — Qwen3.8-27B benchmarks on Strix Halo · 27B and Flash Next on 128GB Strix Halo (r/StrixHalo): same-box Strix Halo proof and serving-config intel
• antirez — gguf-tools: GGUF layout reference
• PaoAI — huggingface.co/PaoAI · paoai-strix-engine · paoai-strix-recipes: the STRIX recipe (per-tensor cut/keep table), imatrix-guided builds, the three profiles, the draft word list, the KV row pad, sweep benchmarking, chain test

Apache-2.0, same as the base model. Not affiliated with Alibaba or the Qwen team.