Qwen3.6-35B-A3B-VQ-5.4bpw

22.2 GiB — the closest to bf16 in this collection.

A vector-quantized build of Qwen3.6-35B-A3B for Apple Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the checkpoint as model.py.

Changelog

2026-10-03 — MTP draft head

This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction draft head for speculative decoding. The weights and the bundled model.py are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16 release (stripped from the common MLX conversions), quantized to 6-bit, group 32. Knurlogic drafts with it automatically; see Speculative decoding. The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.

2026-09-17 — the full Qwen3.6-35B-A3B VQ ladder on one KL instrument

KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.

Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.

Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

The bundled model.py runtime is updated as part of the 2026-09 refresh shared across the VQ lineup:

• Updated MoE expert kernels (uint32 code loads for the d2 geometry, template-free specialized dispatch). For the code geometries this model uses, outputs are verified equivalent to the previous bundle and the speed change is modest; the refresh is mainly for runtime consistency across the lineup.
• The bundle loads under both mlx-lm and mlx_vlm.
• Nothing you have downloaded breaks: weights are unchanged, only model.py is replaced, and the prior revision stays pinnable by commit hash.

Measured results

See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.

Referee corpus, 2048-token windows, scored against the bf16 teacher on the same files with an unmodified mlx-lm. All sizes include the bf16 vision tower, which every build here carries:

This build sits between the two community rungs and closer to the 8-bit than to the 4-bit — 28.1 millinats against their 78.6 and 7.4. Interpolating the affine frontier between the 4-bit and a 6-bit build to this exact size gives 38.7 millinats; this build measures 28.1, a factor of 1.4 below the line.

Two independent fits of this recipe were scored, at 28.141 and 27.927 millinats — the only build in this collection measured twice. The spread between them, 0.214 millinats, is this family's fit-to-fit noise floor, and the margin above is roughly fifty times it. The number quoted in the table is one artifact, the one published here, not an average of the two.

Perplexity was not measured on this artifact. The other releases in this family carry a perplexity column; this build was scored on KL and top-1 agreement, which is the instrument this family is ranked by. Reporting a sibling's perplexity here would be a substitution this project does not make.

Runtime

Speed on this revision (measured 2026-10-01)

One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run, alternating with the reference, 3 pairs, 2048-token prompt, 128 generated tokens, no drafting.

The reference's own decode speed swung between pairs in this run, so treat the decode ratio as approximate.

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Comfortable on a 32 GB machine.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-5.4bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-5.4bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-5.4bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Speculative decoding (MTP)

Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside the weights (a full download includes it), and switches between drafting and plain decoding by measured cost.

• Turn it off: knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in the load panel.
• Force it on, without automatic switching: --set KNURLOGIC_MTP_DYNAMIC=off.
• Requires Knurlogic 0.1.3 or newer (pip install -U knurlogic).

Measured on this build with this head (M4 Max, one Mac, greedy, thinking off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-5.4bpw \
  --prompt "Write a Python function that ..." \
  --max-tokens 512

Methodology

Uniform vector quantization of the experts at d=2, K=1024 — each 2-weight subvector stores one 10-bit index into a per-tensor 1024-entry fp16 codebook, with an fp16 scale per (row, 64 weights). Non-expert tensors are 8-bit.

Halving the subvector dimension is what buys the quality here. At d=4 every additional quarter-bit costs a doubling of the codebook, so the d4 line saturates; spending the same bits at d=2 keeps paying. There is no depth schedule and no mixed allocation — every expert tensor uses the same geometry.

Codebooks are fit in pure weight space — k-means over the weight subvectors, no Hessian, no activation statistics, no calibration corpus.

The fit is not seeded. k-means draws an unseeded subsample, so this artifact is reproducible in recipe and geometry but not bit-for-bit. That is why two fits were scored rather than one.

Verification

Every tensor was decoded from the published artifact and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error (median 0.0411). The bundled runtime was exercised as the executing copy in a stock venv, not merely present in the folder.

Vision tower grafted in (333 tensors, 0.83 GiB); the text path is unchanged, verified against the same KL cache.

Limitations

• The 35 GiB 8-bit is still measurably closer to bf16 (7.4 millinats against 28.1). This is the closest VQ build here, not a lossless one.
• No perplexity, no throughput, and no task-suite scores for this artifact — see above.
• One asymmetry favors this build: the affine comparators quantize the MoE router, and this build leaves it at bf16. The bytes are trivial, but a router feeds an argmax over experts, so the effect is not bounded by the byte share. It is unmeasured.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.