Qwen3.8-9B Heretic Uncensored - 4-bit MLX + MTP (MTPLX)
Fast local build of rohit267/Qwen3.8-9B-heretic-uncensored for Apple Silicon, tuned for native-MTP speculative decoding with MTPLX.
Measured 21.25 -> 40.05 decode tok/s (+88%) on an M5 24GB MacBook, lossless (exact speculative verify - the emitted tokens match the target model's own decoding).
Contents
| File | What it is |
|---|---|
model.safetensors (+ index) | 4-bit affine MLX trunk, group size 64 (4.5 bpw, ~5.0 GB) |
mtp.safetensors | Native MTP head sidecar (Qwen3.5-9B lineage) |
mtp-adapter-r64.npz | Trained C4 LoRA MTP adapter (rank 64) that lifts draft acceptance on this fine-tune |
mtplx_runtime.json | MTPLX runtime contract (depth, hidden variant, sampler) |
The adapter is merged at load time; it is not baked into mtp.safetensors, so the
base sidecar stays reusable.
How it was built
- Convert the BF16 source to 4-bit MLX:
mlx_lm.convert --q-bits 4 --q-group-size 64. - Graft the Qwen3.5-9B MTP sidecar, then train a rank-64 C4 LoRA adapter on hidden states captured from this 4-bit trunk (coding + math + long-code calibration). Training the adapter on the final trunk is required - an adapter calibrated on a different trunk pairs wrongly.
- Serve with a tape-replay GDN verify kernel at draft depth 2.
Full lever study (every quantization / depth / adapter-rank / verify-core tested): see the autoresearch repo linked below.
Run it
Requires macOS 14+, Apple Silicon, and MTPLX (brew install youssofal/mtplx/mtplx).
mtplx serve \
--model . \
--model-id qwen-heretic \
--mtp --depth 2 \
--profile sustained \
--verify-strategy capture_commit \
--verify-core linear-gdn-from-conv-tape \
--mtp-adapter mtp-adapter-r64.npz --merge-mtp-adapter \
--host 127.0.0.1 --port 8080
Serves OpenAI /v1/chat/completions and Anthropic /v1/messages (Claude Code / OMP compatible).
Notes / caveats
- Apple Silicon + MLX only. The MTP fast path is MTPLX-specific.
- Lossy trunk, lossless decode. 4-bit quantization changes the weights; the speculative decode itself is exact-verify, so output matches the 4-bit target model.
- This fine-tune always opens with a
<think>block; budget tokens accordingly. - Draft acceptance on the benchmark suite is ~0.71 at depth 2; depth 3+ does not pay on this hybrid (GatedDeltaNet) 9B. ~40 tok/s is the honest lossless ceiling on this model / this hardware.
License
Apache-2.0, inherited from the Qwen3.5-9B base.