Qwen3.8-27B, MLX 4-bit for mlx-serve

4-bit MLX conversion of Qwen/Qwen3.8-27B, built for mlx-serve, the native Zig MLX server for Apple Silicon (mlxserve.com · github.com/ddalcu/mlx-serve).

18.2 GB on disk, so the whole 27B fits comfortably on a 32 GB Mac with room for a long context. If you have more, take the 8-bit build instead.

Which one

Measured on these weights, quantization noise per matmul-read layer (RMS error over the layer's own RMS, median across attention, MLP and lm_head at layers 0 / 20 / 42 / 63):

This build is the one that fits a 32 GB Mac. On a machine that holds 8-bit, that one is 13x quieter for 1.7x the bytes and about 1.4x the decode time. Text, vision, tool calling and thinking all work, and the checkpoint's native MTP head is included and served, which is worth up to 2.9x on decode (numbers below).

What mlx-serve is

A single native binary that runs any LLM on your Mac. No Python, no cloud, no Electron. It speaks OpenAI-compatible and Anthropic-compatible HTTP on the same port, so Claude Code, the OpenAI SDK, Continue, Cursor, Open WebUI and anything else that talks one of those wires just works against http://localhost:11234. Beyond text it also generates images, video, music, speech with voice cloning, and 3D models, all natively on MLX in the same server.

It ships with MLX Core, a signed and notarized macOS menu-bar app with chat, agent mode, MCP tool calling, model browsing and downloads.

• Site: mlxserve.com
• Source and releases: github.com/ddalcu/mlx-serve
• Download the app: MLX Core.app

Speed on mlx-serve

M4 Max (128 GB), macOS 26.5, temperature 0, median of 3, taken from the server's own reported decode rate. This is mlx-serve against itself with the checkpoint's MTP head on and off, not a comparison against another engine:

Code is where multi-token prediction pays: indentation, closing brackets and repeated identifiers are all predictable enough to draft several ahead. Prose accepts fewer drafts and gains proportionally less. Drafted tokens are verified against the trunk, so the output is the model's own either way.

Serving

mlx-serve --model ddalcu/Qwen3.8-27B-MLX-Serve-4bit --serve

Or pull it from the model browser in MLX Core and hit Load.

• Thinking is on by default. Turn it off per request with "enable_thinking": false, or tune the depth with "reasoning_effort": "xhigh" | "medium" | "low" (Qwen3.8's own vocabulary, default xhigh). Reasoning comes back as reasoning_content on both the OpenAI and Anthropic APIs.
• Tools use Qwen3.8's XML call format (<tool_call><function=name><parameter=key>). mlx-serve parses, repairs and schema-coerces it into standard OpenAI tool_calls.
• Images work on the chat APIs. Send image_url content as usual.
• Context is 262,144 tokens natively; mlx-serve sizes the window to your machine at load.

Conversion

• Affine 4-bit, group size 64, on every matmul-read weight: attention q/k/v/o (the q projection carries the fused output gate), the MLPs, the GatedDeltaNet in/out projections, and lm_head.
• Kept bf16: embed_tokens (a gather-read table, not a matmul), the whole vision tower, mtp.fc, and every norm, bias, conv and SSM state.
• The MTP head ships with the model. mlx-serve finds it in the trunk shards and loads it with no flags.

Three things had to be fixed to make the raw checkpoint serve correctly on MLX, noted here because anyone converting this model themselves hits all three:

1. The norms are delta-encoded. Qwen3.8 stores the decoder layer norms, the final norm and q/k_norm zero-centered, so the layer computes 1 + w. The +1 is baked into this conversion. linear_attn.norm and the vision tower are not delta-encoded and are left alone. Miss this and the model emits pure gibberish from the first token.
2. The depthwise conv1d needs transposing. PyTorch ships [C, 1, K], MLX reads [C, K, 1].
3. The Conv3d patch embed is channels-last in MLX. PyTorch's [out, C, T, H, W] becomes [out, T, H, W, C]. The shapes are the same size either way so nothing errors, the tower just reads the colour channels as spatial and describes a striped image. It still gets shapes and positions right, which makes this one easy to miss.

Weights, config, tokenizer and chat template are otherwise verbatim from the base repo.