Qwen3.6-35B-A3B-VQ-3.4bpw

13.8 GiB (vision tower included) — beats the 19 GiB 4-bit on perplexity at 73% of its size.

The compact build in this collection: a vector-quantized Qwen3.6-35B-A3B for machines where the 18.7 GiB parity build is too large. Stock mlx-lm.

Changelog

2026-10-03 — MTP draft head

This repo now ships mtp-head-q6.safetensors, a 0.70 GiB multi-token-prediction draft head for speculative decoding. The weights and the bundled model.py are unchanged. The head is Qwen3.6-35B-A3B's own mtp.* layer from the bf16 release (stripped from the common MLX conversions), quantized to 6-bit, group 32. Knurlogic drafts with it automatically; see Speculative decoding. The exo --mtp note in the 2026-09 entry below is superseded: exo is no longer the serving path, and no head shipped until this revision.

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac, drafting and vision instructions now use Knurlogic.

2026-09-17 — the full Qwen3.6-35B-A3B VQ ladder on one KL instrument

KL to this family's own bf16 teacher (cached top-64), 12288 tokens, on all three house corpora — prose, public code and literary — every rung scored on the same cache and the same positions, through a streamed pass validated against a direct full-model forward to all printed decimals.

Millinats per token; lower is closer to bf16. These rows are VQ rungs only and are not comparable to the affine-comparator table further down, which was measured prose-only at 2048 tokens on an earlier instrument.

Rank by KL, not perplexity. On this family perplexity is not monotone in quality: rungs that are measurably closer to the teacher can read a higher perplexity, because a damaged model can score better than bf16 on a finite sample. KL on cached teacher logits does not have that failure mode.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

The bundled model.py runtime is updated as part of the 2026-09 refresh shared across the VQ lineup:

• Updated MoE expert kernels (uint32 code loads for the d2 geometry, template-free specialized dispatch). For the code geometries this model uses, outputs are verified equivalent to the previous bundle and the speed change is modest; the refresh is mainly for runtime consistency across the lineup.
• The bundle loads under both mlx-lm and mlx_vlm.
• Nothing you have downloaded breaks: weights are unchanged, only model.py is replaced, and the prior revision stays pinnable by commit hash.

Measured results

See also the 2026-09-17 entry in the changelog above, which places this rung against the rest of its family on the three-corpus KL instrument. The table below is the earlier prose-only measurement and is not comparable to it row-for-row.

+2.9% perplexity against the 4-bit's +4.1%, at 73% of its size — and the gap widens on a code corpus (+1.9% vs the 4-bit's +4.6%, same harness).

This is a genuine quality step down from the parity build — it costs ≈3.8% perplexity to save 4.9 GiB. If you have the RAM, take the 18.7 GiB one.

Perplexity is corpus-specific; compare only against models scored on the same files.

Runtime

Speed on this revision (measured 2026-10-01)

One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run, alternating with the reference, 3 pairs, 2048-token prompt, 128 generated tokens, no drafting.

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Earlier figures (previous runtime)

Single M3 Ultra, macOS, stock mlx-lm, GPU otherwise idle:

Measured on an M4 Max (128 GB), mlx-lm, 120-token greedy generation.

Runs on a 16 GB machine at short context; comfortable on 24 GB+.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.4bpw --local-dir ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.4bpw
knurlogic serve ~/Knurlogic/Models/Qwen3.6-35B-A3B-VQ-3.4bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Speculative decoding (MTP)

Knurlogic drafts with mtp-head-q6.safetensors by default when it sits beside the weights (a full download includes it), and switches between drafting and plain decoding by measured cost.

• Turn it off: knurlogic serve <model> --set KNURLOGIC_MTP=off, or MTP Off in the load panel.
• Force it on, without automatic switching: --set KNURLOGIC_MTP_DYNAMIC=off.
• Requires Knurlogic 0.1.3 or newer (pip install -U knurlogic).

Measured on this build with this head (M4 Max, one Mac, greedy, thinking off, 400 tokens, 3 prompts; Knurlogic main + DSpark work, mlx 0.32.3):

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/Qwen3.6-35B-A3B-VQ-3.4bpw \
  --prompt "Explain the difference between a mutex and a semaphore." \
  --max-tokens 512

Methodology

Uniform vector quantization of the experts at d=4, K=2048 (3.0 bits/weight packed); non-expert tensors at 8-bit. Experts are ≈92% of the model, so that is where the bytes are.

Raising K further does not pay: K=4096 bought +0.55 agreement points for +0.9 GiB, which is why this rung stops here.

Verification

Every tensor decoded from the published artifact and compared against the bf16 source; nothing exceeds 3x the artifact's own median reconstruction error. Packing verified as a pure representation change — packed and unpacked score identically to three decimals.

Limitations

• Measurably below the 18.7 GiB sibling (1.029x vs 0.991x ppl) and well below the 35 GiB 8-bit. This is the size-first choice, not the quality choice.
• Perplexity measured on one corpus; a different workload may rank builds differently.
• No blind human-preference evaluation was run on this model.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: hello@thedrainflorist.com

Provenance

Base model: Qwen/Qwen3.6-35B-A3B — Apache-2.0. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.