Qwen3.8-27B-Abliterated — MLX bf16 with native MTP
Unquantized MLX conversion of windowsxp811203/Qwen3.8-27B-Abliterated (an abliterated Qwen/Qwen3.8-27B) with the MTP draft head and the vision tower kept in the checkpoint. 51.75 GiB (55.56 GB) of safetensors — this is the reference the quantized builds were made from, and the right starting point if you want to run your own oQ / mlx-vlm quantization with MTP preserved.
| runtime | what you get |
|---|---|
| oMLX | native MTP speculative decoding (mtp_enabled: true) + vision, at full bf16 quality |
| mlx-vlm 0.6.17 (tested on the oQ siblings) | loads as a normal bf16 VLM — its sanitize() drops the in-file MTP tensors; add --draft-model <the downloaded drafter/ folder> for MTP. Releases that skip sanitize on MLX-format files (e.g. 0.6.3) reject the language_model.mtp.* keys |
Quantized siblings (same conversion, oMLX oQ with preserve_mtp=True — MTP head kept, its Linears at the base bits, mtp.fc left bf16): oQ4e/ ((15.81 GiB)) · oQ6e/ ((22.09 GiB)) · oQ8e/ ((27.94 GiB)) — all in the project repo.
What is in the file
- 1199 tensors, all bf16: language model (64 layers, 48 GatedDeltaNet + 16 full attention), vision tower 333 tensors,
MTP head 15 tensors (
language_model.mtp.*,mtp_num_hidden_layers: 1),lm_headuntied. - MLX conventions applied by the converter: the zero-centered RMSNorm weights carry the +1 offset (input/post-attention
layernorms,
q_norm/k_norm, the final norm and every MTP norm includingpre_fc_norm_*; the GDNlinear_attn.normgated RMSNorm is left as-is), the GDNconv1dis transposed to(channels, kernel, 1), and the visionpatch_embed.proj.weightis moved to channels-last(out, T, H, W, in). - Provenance: converted with
mlx_vlm.convert(quantize=False, dtype=bfloat16)under oMLX 0.6.4's MTP sanitize patch (mlx-vlm 0.6.3 @ 78b96eb, mlx 0.32), straight from the HF bf16 parent with no dtype change (all 1199 tensors bf16); apart from the three MLX layout conventions above, tensors are copied unchanged.
Verification
- Probe against the HF parent: 15/15 MTP and 333/333 vision tensors present, all 1199 tensors bf16, and the sampled RMSNorm offsets (7 MTP + 3 trunk norms) are +1.000 within bf16 rounding.
- MMLU (oMLX accuracy bench: 5-shot, its seeded 400-question sample stratified by subject, temperature 0, non-thinking, MTP enabled at depth 3 — the same settings as the oQ runs; single-letter answers barely exercise the draft path, so the score is effectively independent of MTP): 82.50 % (330/400) — the reference the oQ cards compare against.
- Refusal was not re-run on this bf16 build. The parent's numbers carry over (same weights, MLX layout): AdvBench 520 → 1/520 (0.19 %) non-thinking, 0/520 thinking; HarmBench safety categories → 0/293. The oQ4e/oQ6e/oQ8e siblings quantized from it each measured 0/80 on AdvBench and 0/119 on HarmBench-safety (no prefill, non-thinking).
Usage (oMLX)
brew tap jundot/omlx https://github.com/jundot/omlx && brew install jundot/omlx/omlx
hf download windowsxp811203/Qwen3.8-27B-Abliterated-MLX-MTP --include "bf16/*" --local-dir ~/.omlx/models
omlx serve --model-dir ~/.omlx/models # the model id is the folder name: bf16
Then in ~/.omlx/model_settings.json (note the models wrapper — a top-level model key is silently ignored):
{"version": 1, "models": {"bf16": {"mtp_enabled": true, "mtp_num_draft_tokens": 3}}}
The server log should print Speculative backend selected … Lightning MTP (model_type=qwen3_5, active) when it loads.
Support / 打賞
If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。
USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888
Disclaimer
Inherits the Apache-2.0 license of the base model. This model will not refuse; you are responsible for your use.