mlx-community/Qwen3.8-27B-OptiQ-4bit
Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs
A 4-bit mixed-precision MLX quant of Qwen/Qwen3.8-27B, an image+text model. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit.
The vision tower is kept at bf16 in a sidecar, so this repo loads text-only under stock mlx-lm and image+text under OptiQ.
Quantization details
| Property | Value |
|---|---|
| Predominant precision | 4-bit |
| Layers at 8-bit (sensitive) | 261 |
| Layers at 4-bit (robust) | 237 |
| Total quantized layers | 498 |
| Group size | 64 |
| Vision tower | bf16, optiq/optiq_vision.safetensors (0.9 GB) |
| Calibration mix | six-domain mix (40 samples × 6 domains) |
| Reference for sensitivity | uniform-4-bit (bf16 at this size does not fit 36 GB) |
| On-disk size | 19.0 GB |
Qwen3.8-27B is a hybrid: three quarters of its 64 blocks use linear attention (five projections each) and the rest full attention, which is why the sweep scores 498 tensors rather than the ~450 a dense model of this size would have.
We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average.
Usage
Load it with mlx-lm for text:
pip install mlx-lm
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Qwen3.8-27B-OptiQ-4bit")
response = generate(
model, tokenizer,
prompt="Explain quantum computing in simple terms.",
max_tokens=200,
)
For image+text, mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, an OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters and sandboxed Python execution, install mlx-optiq:
pip install mlx-optiq
optiq serve --model mlx-community/Qwen3.8-27B-OptiQ-4bit
See the Qwen family guide on mlx-optiq.com for sampling defaults, training recipes, and family-specific caveats.
Benchmarks
| Metric | Score |
|---|---|
| MMLU (5-shot, 1000) | 84.1% |
| GSM8K (3-shot CoT) | 96.1% |
| IFEval (prompt-level, strict) | 77.1% |
| BFCL-V3 simple (AST) | 93.5% |
| HumanEval (pass@1) | 92.1% |
| HashHop (long-context retrieval) | 85.0% |
| Capability Score (mean of 6) | 87.98 |
18.9 GB on disk.
Every metric gets one equal vote, and disk size is reported next to the score as an honest second axis rather than folded into it. See the eval-framework writeup for the methodology.
Links
- Project website: mlx-optiq.com
- PyPI: pypi.org/project/mlx-optiq
- Calibration mix: mlx-optiq.com/blog/calibration-mix
- Eval framework: mlx-optiq.com/blog/eval-framework
- Base model: Qwen/Qwen3.8-27B
Quantize your own
This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:
pip install mlx-optiq
optiq convert Qwen/Qwen3.8-27B --target-bpw 5.0 --candidate-bits 4,8
optiq lab # or drive the whole thing from the browser