Qwen3.6-35B-A3B — vram13: an all-VRAM ultra quant (12.98 GB, imatrix)

A 35B MoE that fits entirely in 18 GB of VRAM and decodes at 122–159 t/s on a pair of mid-range gaming GPUs (RTX 3060 Ti 8 GB + RTX 3080 10 GB), with quality within ~1 % perplexity of Q8_0.

Most sub-24 GB rigs run 35B-A3B MoE models with CPU-offloaded experts and hit a hard RAM-bandwidth wall (~75 t/s on dual-channel DDR4, no matter the GPU). This quant takes the other exit: shrink the model until every weight lives in VRAM. No offload, no -ot, no RAM wall.

Files

Measured performance (RTX 3060 Ti + RTX 3080, ik_llama.cpp, q4_0 KV)

Decode holds >100 t/s even 14.5 K tokens deep. VRAM use: ~6.5 GB + ~8.4–9.4 GB across the two cards.

Quality (perplexity, held-out corpus, ctx 512)

The imatrix + selective layer protection recovers more than half the quality gap of a naive q2 build at nearly the same size.

Recipe

Requantized from Q8_0 with an imatrix (llama-quantize --allow-requantize):

• Experts (the bulk of the bytes): q2_K on middle layers 3–36, q3_K on the more sensitive first/last three layers (0–2, 37–39)
• Attention + SSM projections: q5_K (this hybrid arch — attention + gated-delta-net SSM — is sensitive here, and these tensors are small)
• Shared expert, output tensor, embeddings: q6_K
• imatrix on everything — calibrated on a mixed code/prose corpus

The principle: on a MoE with small active experts, expert weights dominate size but tolerate 2-bit with imatrix guidance; everything that touches every token (attention, SSM, shared expert, output) stays at 5–6 bit where the byte cost is negligible.

The MTP variant

Qwen3.6's official GGUFs ship without the MTP head, but Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF (an Apache-2.0 coding fine-tune of the same base) kept its trained head. That head was grafted onto this quant by GGUF surgery — append the blk.40 nextn tensors, bump block_count 40→41, set qwen35moe.nextn_predict_layers=1.

Self-speculative decoding verifies every draft against the target model, so output is lossless — only the acceptance rate (speed) depends on how well the donor head predicts this model. On code it accepts well: +30 % decode over the base quant.

Running

Needs ik_llama.cpp (mainline llama.cpp works for the base file; the -mtp flag for the MTP variant is ik-specific).

# base — full 256K window
llama-server -m Qwen3.6-35B-A3B-vram13-q2ex-imat.gguf \
  --jinja -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
  --ctx-size 262144 -ngl 99 --tensor-split 44,56 --ubatch-size 256 --no-mmap

# MTP — fastest for code (128K is the MTP ceiling)
llama-server -m Qwen3.6-35B-A3B-vram13-q2ex-imat-MTP.gguf \
  --jinja -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
  --ctx-size 131072 -ngl 99 --tensor-split 44,56 --ubatch-size 256 --no-mmap \
  -mtp --ctx-size-draft 8192

Single 16 GB+ GPU: drop --tensor-split. On 2×8 GB it will not fit — use a CPU-offload quant instead.

Qwen3.6 is a reasoning model: it emits chain-of-thought into reasoning_content and the final answer into content — give it generous max_tokens, or disable thinking with "chat_template_kwargs": {"enable_thinking": false}.

Caveats

• 2-bit experts are not free: expect slightly weaker edge-case reasoning than Q4-class quants. For maximum quality at lower speed, use a Q4/mixed quant with CPU offload.
• The MTP head is from a fine-tune of this base, not this exact model — lossless, but acceptance (speed) may vary by domain; prose gains are smaller than code.
• Built and tested on ik_llama.cpp (July 2026). The MTP file declares 41 blocks; loaders that don't understand nextn_predict_layers should ignore the extra tensors, but mainline behavior isn't guaranteed.

Built with TurboQuant/ik_llama.cpp tooling on a Ryzen 5950X + 3060 Ti + 3080. Base model © Alibaba Cloud, Apache-2.0.