Qwen3.8-Flash-Next-Abliterated-MTPLX-4bit
Abliterated 4-bit build of Qwen3.8-Flash-Next optimized for MTPLX
Updated 2026-08-31 — please re-download
If you downloaded this model before 2026-08-31, you have v1. Re-download for roughly a 10% speed boost.
v2 is forged directly from the abliterated BF16 weights rather than repackaged from the oQ4e MLX build. Same size, same architecture, same sidecars.
| v1 Original | v2 (live version) | delta | |
|---|---|---|---|
| Recommended: MTP, draft_depth 1 | 38.0 – 61.6 tok/s | 41.1 – 67.4 tok/s | +8-9% |
| Short, thinking xhigh, temp 1, MTP | 45.1 | 47.9 | +6.2% |
| Long agentic (26.5k), thinking xhigh, temp 1, MTP | 38.7 | 43.6 | +12.7% |
| Fastest single cell: short, temp 0, MTP | 61.6 (thinking low) | 67.4 (thinking medium) | +9.4% |
| Short, thinking xhigh, temp 1, AR | 34.9 | 48.2 | +38.1% |
| Long agentic, thinking xhigh, temp 1, AR | 31.0 | 43.5 | +40.3% |
| Size on disk | 115.7 GB | 116.0 GB | +0.3% |
Measured on the same Mac Studio M3 Ultra, same prompts, same harness, both on MTPLX 2.10.1. v2 wins all 41 comparable cells across three fixtures, both modes, all thinking levels, both temperatures.
Lineage: Qwen/Qwen3.8-Flash-Next → Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16 (abliterated) → this build, forged for MTPLX.
| Abliteration (upstream BF16, Qwentium persona) | 2.3% refusal on harmful prompts (7/300), 2.2% full suite (10/450) |
Intended use and terms
Published for experimental AI research and AI-safety research only: red-teaming, refusal and alignment research, interpretability, and quantization or inference research.
It must not be used for anything illegal, in any jurisdiction, under any circumstances. The removal of refusal behavior is a research property of the artifact, not an invitation, a permission, or an endorsement. Nothing about publishing this model condones or excuses illegal activity.
You are solely responsible for your use of it and for complying with applicable laws and upstream licence terms. It ships with essentially no safety refusals: assume it will comply with any instruction, including harmful ones. Run it locally on hardware you control. Do not expose it as a public endpoint or deploy it to untrusted users.
Will it fit on my Mac?
There are two parts to load: the model itself (84.0 GB) and a 32.0 GB n-gram table that helps it run faster. The model always sits in memory. The n-gram table is the part you have a choice about.
| memory needed | |
|---|---|
| n-gram table kept in memory (faster) | ~116 GB |
| n-gram table read from SSD | ~84 GB |
You do not have to configure this. On a Mac with 160 GB or more of unified memory, MTPLX keeps the n-gram table in memory by itself. On any Mac with less than that, it reads the table from your SSD instead and the model still runs normally, roughly 12% slower.
To override the automatic choice, MTPLX_NGRAM_RESIDENT=1 pins the n-gram table
in memory and MTPLX_NGRAM_RESIDENT=0 pins it to SSD. Unset means automatic, so
on a large Mac you need =0 to stream.
Quick start
Apple Silicon. Resident n-gram wants ~116 GB of unified memory; streaming the table from SSD instead needs ~84 GB plus a hot cache.
pip install mtplx
hf download grant-ai/Qwen3.8-Flash-Next-Abliterated-MTPLX-4bit \
--local-dir ~/models/qwen38-abliterated-mtplx
MTPLX_NGRAM_RESIDENT=1 mtplx serve \
--model ~/models/qwen38-abliterated-mtplx \
--model-id qwen38-abliterated \
--generation-mode mtp --depth 1 \
--port 8000
On a Mac with less than ~160 GB of RAM, drop MTPLX_NGRAM_RESIDENT=1. MTPLX
streams the n-gram table from SSD automatically below that threshold and the
model still runs, about 12% slower.
Then use the OpenAI-compatible endpoint:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "qwen38-abliterated",
"messages": [{"role": "user", "content": "Explain MoE routing."}],
"max_tokens": 1024, "temperature": 1.0, "top_p": 0.95, "top_k": 20}'
--generation-mode mtp --depth 1 is the recommended configuration. It is
substantially faster than ar at temperature 0 and at short context. At long
context with temperature 1 the two are level. See Speed below.
Speed
Hardware: Mac Studio M3 Ultra, 32-core CPU, 80-core GPU, 256 GB unified
memory, fans pinned to max. MTPLX 2.10.1 stock, measured 2026-08-31 at
--depth 1. Decode throughput, engine-reported, median of 2 reps at temperature
0 and 3 at temperature 1.
Short context (289 tokens)
AR baseline: 42.9 to 44.5 tok/s at temp 0, 48.2 to 49.8 tok/s at temp 1.
| thinking level | decode tok/s (temp 0) | gain over AR | decode tok/s (temp 1) | gain over AR |
|---|---|---|---|---|
| off | 59.9 | +34.6% | 54.9 | +13.9% |
| low | 66.0 | +53.5% | 54.8 | +10.0% |
| medium | 67.4 | +56.4% | 59.5 | +22.4% |
| xhigh | 60.6 | +41.3% | 47.9 | -0.6% |
Long agentic context (26,531 tokens)
AR baseline: 38.8 to 39.3 tok/s at temp 0, 43.4 to 43.5 tok/s at temp 1.
| thinking level | decode tok/s (temp 0) | gain over AR | decode tok/s (temp 1) | gain over AR |
|---|---|---|---|---|
| off | 43.4 | +10.4% | 41.1 | -5.5% |
| low | 43.1 | +10.8% | 42.5 | -2.1% |
| medium | 43.9 | +12.6% | 42.6 | -1.8% |
| xhigh | 43.4 | +11.9% | 43.6 | +0.2% |
At long context the speed does not depend on how hard the model thinks. All
eight cells land between 41.1 and 43.9 tok/s. Reasoning at xhigh over 26k
tokens of context costs nothing against thinking off.
Speculative decoding pays at temperature 0 and at short context. At long context with temperature 1 it is level with plain autoregressive decoding: the baseline is fast enough that drafting has little left to win.
Draft depth
Measured at long agentic context, thinking xhigh, temperature 1.
| draft depth | decode tok/s | tokens per draft step |
|---|---|---|
| 1 | 42.3 | 1.53 |
| 2 | 40.9 | 1.79 |
| 3 | 37.6 | 1.81 |
| 4 | 32.1 | 1.76 |
| 5 | 28.2 | 1.77 |
| 6 | refused by the engine |
Deeper drafting commits more tokens per step and is still slower: the extra draft work costs more time than the extra accepted tokens return.
Vision
Vision works. The 333 vision tensors ship at fp16 in this pack and the tower is constructed by MTPLX's serving path, not by the text backend. Re-verified on 2.10.1 with a live image request.
Requires
--generation-mode mtp. Inarmode the server refuses image content outright. That is a clean refusal, not a silent failure.
Abliteration
Abliteration has not been remeasured on this build. The figures are upstream BF16, carried through quantization unchanged.
| refusal rate, harmful prompts | 2.3% (7 of 300) |
| refusal rate, full suite | 2.2% (10 of 450) |
| persona measured | Qwentium |
Personas
Ships with the stock Qwen chat template as the default, which injects no system prompt. The abliterated weights carry the behaviour on their own.
The upstream Blackfrost "Qwentium" persona is included as an opt-in: a ~1,200 token compliance-oriented system prompt prepended to every request. Switch with the bundled script, then restart your server:
bash switch_persona.sh blackfrost # persona on
bash switch_persona.sh qwen # back to stock (default)
bash switch_persona.sh status # which is active
The template lives in two places, chat_template.jinja and the chat_template
key inside tokenizer_config.json, and different runtimes read different ones,
so the script switches both. It is safe against an hf download cache checkout:
it replaces symlinks rather than writing through them.
Settings that matter
| setting | value |
|---|---|
| sampler | temperature 1.0, top_p 0.95, top_k 20 (from the model's own generation_config.json) |
| generation mode | --generation-mode mtp --depth 1 |
| n-gram residency | MTPLX_NGRAM_RESIDENT=1. Auto-arms at >=160 GB RAM |
| QSA gather | MTPLX_QSA_GATHER=1 MTPLX_QSA_GATHER_DECODE=1 MTPLX_QSA_GATHER_MAX_ROWS=16 adds about 2% at long context and costs about 1% at short. Off by default in MTPLX 2.10.1 |
| MTP depth | 1. Depth 3 never wins; depths 4 and 5 are slower than not speculating |
| thinking | chat_template_kwargs: {"enable_thinking": false} to disable |
| reasoning effort | --reasoning-effort low | medium | xhigh, or reasoning_effort in the API |
Technical details
| architecture | qwen4_exp, hybrid Gated-DeltaNet + Qwen Sparse Attention, MoE, PLE n-gram embeddings |
| quantization | 4-bit group 32 base; 8-bit group 64 on embed_tokens, lm_head, linear_attn.out_proj, mlp.gate, shared_expert_gate, shared_expert.{gate,up,down}_proj and indexer.index_qk_proj; hyper-connection mixers and ple.{key,value}_proj kept bf16 |
| bit widths | 4 and 8 bit, group sizes 32 and 64 |
| MTP head | 13 tensors quantized: 4-bit g32 on the switch_mlp and self_attn projections, 8-bit g64 on the shared-expert and indexer projections. Shipped as mtp.safetensors (1.68 GB) |
| vision | 333 tensors, fp16, working in mtp mode |
| runtime | MTPLX 2.10.1 (stock) |
| n-gram table | 320,001,536 rows, 4-bit g32, shipped as ngram-table.safetensors (32.0 GB) |
How it was made
- Forged from
Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16
through
mtplx forge buildon MTPLX 2.10.1, using the Optimized-Speed recipe above. Unlike v1, the body is genuinely requantized rather than passed through from an existing MLX build. - mlx-lm ships no
qwen4_expbackend, so a shim exposing MTPLX's own implementation was needed for the convert to load the model at all. - The MTP sidecar was re-sanitized: forge extracts it carrying the source's
fused
mlp.experts.*tensor names, which the MTP module rejects. - The MTP head was quantized to the reference layout, 5.21 GB down to 1.68 GB.
- The n-gram table was quantized to 4-bit group 32 into
ngram-table.safetensors. The bf16 forge lane otherwise leaves it inline at 102.4 GB. - Packaged from a tree containing only the shards the weight index references.
Steps 2 through 6 are not reproducible from
mtplx forge buildalone on MTPLX 2.10.1. Rebuilding this artifact from the recipe requires them.
Not measured: output quality was not compared against v1. The two builds use different quantization recipes, and only speed was benchmarked.