Qwen3.8-27B UD-Q3_K_XL — MLX GPTQ-direct (native MLX, 3-bit dynamic)
Native-MLX port of unsloth's UD-Q3_K_XL dynamic quantization for
Qwen/Qwen3.8-27B. Instead of converting GGUF weights, this build runs the
quantizer search natively in MLX's affine format (group size 64) and writes
final MLX weights directly — no GGUF round-trip, no mlx_lm.convert
re-quantization step.
Perplexity (wikitext-2-raw first 32k tokens, 512-token windows, greedy NLL —
identical math to llama-perplexity):
| Build | PPL ↓ |
|---|---|
| unsloth UD-Q3_K_XL GGUF in llama.cpp | 6.896 |
| this model (GPTQ-direct) | 8.090 |
| MLX imatrix-weighted (no GPTQ) | 8.136 |
| MLX plain RTN (same bit map) | 8.169 |
Best published native-MLX Q3 for this model family to our knowledge. Full methodology and history: ud2mlx repo.
Method
- Bit map taken from unsloth's
Qwen3.8-27B-UD-Q3_K_XL.gguf(per-tensor bits, unsloth's dynamic layer selection). - Importance weighting from unsloth's published imatrix (per-column activation energies from their calibration set).
- GPTQ with real activation Hessians captured on 128×512 = 65,536 calibration tokens (wikitext-2-train-raw) — full-rank for every input dim up to I=17408. Sequential error feedback through each block.
- Per-group affine parameters are fixed before feedback (importance- weighted candidate search with margin headroom), so the emitted weights are exactly what the feedback pass produced — zero re-binning loss.
embed_tokensandlm_headhave no block-input Hessian → RTN with the same affine search (uniform importance). lm_head at 5 bits.
14 GB, 498 quantized tensors, group size 64, avg ~4.4 bits on quantized modules (2-bit MLP-heavy early layers up to 8-bit norms/router).
Usage
pip install mlx-lm
mlx_lm.generate --model jclyons52/Qwen3.8-27B-UD-Q3_K_XL-MLX-gptq-direct \
--prompt "Explain quantum tunneling" --max-tokens 256
Works with any MLX runtime that loads mlx-lm models (oMLX, etc.).
Limitations
- Affine vs IQ-codebook gap. MLX's quantized matmul only supports uniform affine grids; unsloth's ≤3-bit GGUFs use non-linear IQ codebooks (E8-lattice / super-block formats). At 3 bits this costs ~+1.2 PPL vs the GGUF; at 2 bits the gap widens to ~+1.95 (details). This is a runtime-kernel limitation of MLX, not a property of the weights.
- Embed/lm_head are RTN, not GPTQ (no activation Hessians available at those positions in the current capture harness).
- Text-only. The visual tower of the base checkpoint is not included.
- Eval scope: perplexity measured on wikitext-2 only, 32k tokens. Downstream task quality (coding, math, multilingual) is not yet characterized; low-bit dynamic quants typically degrade non-uniformly.
- Not for further training — quantized weights are frozen codes; use the fp16 base for fine-tuning.
- Quality can vary by prompt/domain; the late-window PPL drift visible in sequential evals suggests domain sensitivity typical of low-bit quants.
Calibration data
- GPTQ Hessians: wikitext-2-train-raw, first 65,536 tokens.
- Importance weighting: unsloth's published
imatrix_unsloth.gguf(their calibration mixture).
Credits
- unsloth — dynamic bit maps + imatrix
- mlx-lm — MLX runtime
- Pipeline: jclyons52/ud2mlx
Built 2026-08-28. Base model license inherited from Qwen/Qwen3.8-27B.