Qwen3.8-27B-Uncensored-FP8-MTP
FP8 (block 128×128, E4M3) quantization of an abliterated Qwen3.8-27B, with the vision tower and the working MTP speculative-decoding head both preserved.
What "abliterated" means here. The base model has had its refusal direction removed — it will not decline requests the way the official Qwen3.8-27B does. That modification comes from the base model, not from this repository; this repo contributes the FP8 quantization only. The base is Apache-2.0 and publicly available. Use accordingly.
Why this build
Surveying 148 abliterated/uncensored Qwen3.8-27B repositories in August 2026 turned up exactly one FP8 build, and it was access-gated. Everything else was GGUF, NVFP4 (Blackwell-only), AWQ, or unquantized BF16. On Hopper hardware — where FP8 has native tensor-core support — that left no ungated option.
Only a handful of builds in any format preserve the MTP head, and getting it to actually
work under vLLM required the checkpoint to be in the native fp8 format rather than
compressed-tensors (see below).
Measured against the official build
Same hardware (H100 47GB vGPU), same vLLM 0.27.1, same flags.
Qwen/Qwen3.8-27B-FP8 | this build | |
|---|---|---|
| Thai extraction, exact codepoint (temp 0, 4 runs) | 16/16 | 16/16 |
| Coding tasks — generated code executed against assertions | 4/5 | 4/5 |
| Vision: Thai table, activity names read exactly | 4/4 | 4/4 |
| Vision: numeric cells | correct | correct |
| MTP acceptance rate | 58% | 52% |
| tok/s @ concurrency 1 | 131 | 132 |
| tok/s @ concurrency 8 | 816 | 813 |
| tok/s @ concurrency 16 | — | 1,373 |
The one coding task both models fail is the same one, for the same reason: the model spends its whole token budget reasoning and never emits an answer.
Hardware requirements
FP8 needs tensor-core support to be worth anything. On architectures without it, vLLM either refuses the checkpoint or falls back to a dequantize-then-multiply path that is slower than a 4-bit AWQ build.
| Architecture | Examples | Compute cap. | FP8 tensor cores | Runs this model |
|---|---|---|---|---|
| Volta | V100 | 7.0 | ✗ | no — also lacks BF16 tensor cores, and vLLM has dropped Volta |
| Turing | RTX 20xx, T4 | 7.5 | ✗ | no |
| Ampere | RTX 30xx, A100, A40, A6000 | 8.0 / 8.6 | ✗ | see note below |
| Ada Lovelace | RTX 4090, L4, L40S | 8.9 | ✓ | yes, if VRAM allows |
| Hopper | H100, H200, H20 | 9.0 | ✓ | yes — tested here |
| Blackwell | RTX 50xx, B100/B200, GB200 | 10.0+ | ✓ (plus FP4) | yes |
On Ampere (A40, A100, A6000). These have the VRAM — an A40 has 48 GB, the same as the
H100 47C this was tested on — but no FP8 tensor cores. vLLM can fall back to
Fp8MarlinLinearMethod, which keeps weights in FP8 and dequantizes inside the kernel, so
memory still works out. Whether that path accepts block-wise 128×128 scales (as opposed
to per-tensor or per-channel) is untested here — we had no Ampere card to try it on. If you
are on Ampere, an INT8 W8A16 or AWQ build of the same base is the safer choice.
VRAM
Weights occupy ~30 GB. The rest goes to KV cache, which decides how much context you can actually serve.
| VRAM | Verdict | KV pool at --kv-cache-dtype fp8 |
|---|---|---|
| 24 GB (RTX 4090) | will not load on one card | — |
| 32 GB (V100 32GB) | no — wrong architecture entirely | — |
| 48 GB (A40) | VRAM is fine; FP8 path is not — see note above | — |
| 47–48 GB (H100 47C vGPU, L40S) | works | ~350k tokens — measured on H100 47C |
| 80 GB (H100/H200 80GB) | comfortable | ~600k+ tokens |
2× 24 GB with --tensor-parallel-size 2 | works on Ada | splits weights ~15 GB per card |
At 47 GB the whole 262,144-token context fits for a single request with room for roughly one
more — Maximum concurrency for 262,144 tokens per request: 1.3x. Shorter contexts scale
proportionally: ~20 concurrent requests at 16k each.
Without --kv-cache-dtype fp8, KV for 262k context needs 16.17 GiB instead of ~8.5 GiB and
a 47 GB card will refuse to start.
Serving
vllm serve pramoths/Qwen3.8-27B-Uncensored-FP8-MTP \
--max-model-len 262144 \
--gpu-memory-utilization 0.94 \
--kv-cache-dtype fp8 \
--max-num-seqs 16 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--kv-cache-dtype fp8 is required for 262k context on a 47GB card — fp16 KV needs 16.17 GiB
and will refuse to start. --max-num-seqs must stay below the available Mamba state block
count; the default of 256 fails to boot.
Verify MTP is actually working — it can fail silently:
curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total
A Mean acceptance length of 1.00 means every draft is being rejected. You are paying the
drafting cost for nothing.
What was and was not quantized
Follows Qwen/Qwen3.8-27B-FP8 exactly — the distinction is matmul versus control parameter,
not which component of the model:
- FP8:
q/k/v/o_projandgate/up/down_proj, in the language layers and in the MTP head. - BF16: LayerNorms, gates,
A_log,dt_bias,conv1d, and the gated-DeltaNetin_proj_a/in_proj_b— these govern recurrent state, where quantization error accumulates rather than cancelling. - BF16: the entire vision tower,
lm_head,embed_tokens,mtp.fc.
Reproducing
Scripts, benchmarks, and a writeup of the five silent failure modes encountered along the way: https://github.com/pramoth/qwen38-fp8-forge
Quantization needs no calibration data and no GPU — roughly two minutes of CPU.
Attribution
Qwen/Qwen3.8-27B— original model, Apache-2.0JonathanColetti/Qwen3.8-27B-Uncensored— abliterated baseQwen/Qwen3.8-27B-FP8— the quantization recipe this build copies