Qwen3.8-27B-UD-Q3_K_XL-MTP-MLX
An MLX quantization of
Qwen/Qwen3.8-27B for running locally on Apple Silicon —
the default model for chad, a Claude-Code-style
local coding agent.
Quant
- Params: 27B (dense qwen3_5 hybrid: 64 layers, 48 GatedDeltaNet + 16 full attention)
- Scheme: 3-bit group-64 body (MLP gate/up/down + attention) and 3-bit embeddings, with lm_head held at 5-bit
- Footprint: ~12.1 GB weights + a bundled 4-bit MTP head; 262k native context
- RAM: Built for 24 GB Apple Silicon. chad sizes its context window from the live Metal budget, so expect tens of thousands of tokens rather than the full 262k on a 24 GB box.
The naming follows Unsloth's Dynamic 2.0 convention (UD-Q3_K_XL = a dynamic quant that spends extra bits on the layers that matter) so the scheme is recognizable at a glance. It is not literally a llama.cpp k-quant — this is an MLX group-64 affine quant produced by our own per-module predicate (the bulk low-bit, sensitive layers high), with the bit allocation chosen by measured held-out perplexity per gigabyte across candidate recipes.
Multi-token prediction head
mtp.safetensors is the checkpoint's own trained MTP head, quantized to 4-bit
group-64 and bundled alongside the weights. chad loads it as a sidecar and uses it
for self-speculative decoding: draft several tokens with the head, verify them in
one batched forward, accept by exact rejection sampling — so greedy output is
token-identical to the unspeculated path and sampled output keeps the model's true
distribution at any temperature.
The filename matters. mlx-lm globs model*.safetensors, so mtp.safetensors is
invisible to the base loader — which is the point. Named model-mtp.safetensors it
would be picked up by the base loader, has_mtp_weights would go true in
qwen3_5.TextModel.sanitize, and every norm weight would get a second +1.0 shift:
silent corruption, no error.
Chat template
chat_template.jinja differs from upstream in exactly one line: the default
reasoning_effort is medium, where upstream defaults to xhigh. Pass
reasoning_effort explicitly to override it.
Use it
With chad (auto-downloads this model on first run):
uvx chad-code
Or directly with mlx-lm:
from mlx_lm import load, generate
model, tok = load("nathansutton/Qwen3.8-27B-UD-Q3_K_XL-MTP-MLX")
print(generate(model, tok, "Write a haiku about quantization.", max_tokens=64))
License
Inherits the license of the base model
Qwen/Qwen3.8-27B. Review it before use or redistribution.