Qwen3.8-27B · Activation-Aware Mixed 3.5bpw (MLX)

Run with Rapid-MLX: rapid-mlx serve qwen3.8-27b-mixed-3.5bpw starts an OpenAI- and Anthropic-compatible API on your Mac. · Rapid-MLX benchmark vs Apple's MLX

A multimodal, mixed-precision MLX build of Qwen3.8-27B that brings this 27B vision-language model down to a size that runs comfortably on a 24 GB Apple-Silicon Mac — while keeping 4-bit-level reasoning, knowledge, and code.

▶ Run it on Rapid-MLX (OpenAI-compatible, full continuous batching):

pip install rapid-mlx
rapid-mlx serve rapid-mlx/Qwen3.8-27B-mixed-3.5bpw-MLX --port 8000            # text
rapid-mlx serve rapid-mlx/Qwen3.8-27B-mixed-3.5bpw-MLX --port 8000 --mllm     # + vision

• ~13 GB on disk (avg 3.5 bits-per-weight, mixed 2/3/4-bit) — vs ~16 GB for the standard 4-bit build and ~51 GB for bf16.
• Runs comfortably on 24 GB Macs; easy on 32 GB. (Measured peak ~14.6–16.5 GB — see Runtime footprint.)
• Native multimodal — image + text in, text out (vision tower preserved).
• Serves on Rapid-MLX — text with full continuous batching, or vision with --mllm — on Apple unified memory.

Optional MTP acceleration

This repository also contains its paired native Multi-Token Prediction drafter at mtp/model.safetensors. Recent Rapid-MLX versions can opt into lossless speculative decoding with one self-contained download:

rapid-mlx serve qwen3.8-27b-mixed-3.5bpw \
  --speculative-config '{"method":"mtp","model":"rapid-mlx/Qwen3.8-27B-mixed-3.5bpw-MLX","num_speculative_tokens":3}'

MTP remains off by default because its speed benefit varies by Apple chip. In paired greedy validation, emitted tokens were identical to autoregressive decoding. The Rapid Mac app exposes the same per-model opt-in under Settings → Performance.

The optimized MTP artifact layout builds on MTPLX by Youssof Altoukhi (Apache-2.0).

Why this build

A uniform 3-bit quant of this model loses ~6.6 points of factual knowledge (MMLU-Pro) vs 4-bit. This build recovers essentially all of it by spending bits where they matter: an activation-aware mixed-precision allocation assigns each language Linear 2/3/4-bit according to how much it actually moves the model's outputs, rather than quantizing everything uniformly.

Method note: this is not the AWQ algorithm (which does per-channel scaling at a uniform bit-width). It borrows AWQ's salience metric — activation-weighted quantization error — to drive a mixed-precision bit allocation. See below.

How it was made

1. Run a small English calibration set through the model once, capturing the mean input-activation magnitude of every language Linear (AWQ/GPTQ-style salience).
2. Score each module's quantization error weighted by that activation scale — || (W − dequant(quant_b(W))) · diag(act_scale) || — at bit-widths {2,3,4,8}.
3. Greedy-knapsack allocate bits to hit a 3.5 bpw target, minimizing total activation-weighted error under the size budget.
4. Vision tower kept at 4-bit → multimodal preserved. group size 64, affine.

The knowledge loss concentrates in the MLP blocks (not the output head); the allocation promotes the most salient MLP projections to 4-bit and demotes low-salience ones. 3.5 bpw is the sweet spot — higher gave no further gain.

Evaluation (identical harness, greedy decoding)

Reading: knowledge, reasoning, and code match the 4-bit build; the uniform 3-bit knowledge gap is fully closed. Vision sits at the uniform-3-bit level (~3.6 below 4-bit) — the mixed budget was spent on language, not the vision tower.

vs the full-precision original

Run on an identical subset (same 200 questions / 60 problems / 150 images), bf16, 4-bit, and this mixed build land within measurement noise of each other — at n=200 the 95% CI is ≈ ±7 points, so the 1–3 point spread is individual questions flipping, not a quality gap:

At 13 GB this build is statistically indistinguishable from the bf16 original on knowledge and reasoning — full-precision brains at ~1/4 the size.

Runtime footprint (measured, mlx.get_peak_memory)

Saves ~3.9 GB of peak memory vs 4-bit — enough to move the model down a tier: 4-bit needs a 32 GB Mac (or a tuned iogpu.wired_limit on 24 GB), while this build runs on 24 GB out of the box. Measured ~11 tok/s decode on an M2 Pro (32 GB); load ~5 s. Does not fit 16 GB.

The sweet spot (size vs capability)

Going from 4-bit → uniform 3-bit saves ~4 GB but drops 6.6 MMLU-Pro points. This build sits ~1 GB above uniform 3-bit yet keeps all of that knowledge — the best capability-per-gigabyte point in the family, and the one that turns a 24 GB Mac into a comfortable home for Qwen3.8-27B.

Usage

Serve with Rapid-MLX (OpenAI-compatible API):

pip install rapid-mlx

# text — full continuous batching
rapid-mlx serve rapid-mlx/Qwen3.8-27B-mixed-3.5bpw-MLX --port 8000

# vision — add --mllm (serialized one-request-at-a-time lane)
rapid-mlx serve rapid-mlx/Qwen3.8-27B-mixed-3.5bpw-MLX --port 8000 --mllm

Recent Rapid-MLX also ships a short alias — rapid-mlx serve qwen3.8-27b-mixed-3.5bpw --mllm.

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "default",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "Describe this image."},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}}
  ]}]
}'

Limitations

• Vision ~3.6 pts below 4-bit (MMBench-EN); on par with uniform-3-bit. If you need maximum vision accuracy and have 32 GB, use the 4-bit build.
• Does not fit 16 GB — peak ~14.6 GB text-only. This is a 24 GB / 32 GB build.
• One-shot post-training quantization, no retraining.
• Inherits all capabilities and limitations of the base model.

Attribution & license

• Base model: Qwen/Qwen3.8-27B, Apache-2.0. This derivative is released under the same Apache-2.0.
• Runtime: Rapid-MLX — an MLX inference server for Apple Silicon.
• Quantization method implemented from scratch in MLX (activation-aware mixed-precision); no third-party quantization code included.