Qwen3.8-35B-A3B-Distill APEX GGUF

APEX (Adaptive Precision for EXpert models) quantizations of empero-ai/Qwen3.8-35B-A3B-Distill, covering every deployment scenario from maximum quality down to ~10 GB.

• Quantized by: MrFuzzihead
• Method: APEX — a MoE-aware mixed-precision quantization technique for llama.cpp, by the LocalAI team
• More APEX quants: MrFuzzihead's APEX collection

APEX assigns quantization precision per tensor role (routed expert / shared expert / attention / SSM) and per layer position (edge layers high, middle layers compressed), exploiting the sparsity of Mixture-of-Experts models. On the APEX reference model (Qwen3.5-35B-A3B), APEX tiers match or beat Q8_0 quality at ~60% of the size — see the APEX technical report for details.

Note on the hardware-compatibility widget: Hugging Face detects quantization variants from standard quant names (Q4_K_M etc.) in the filename. APEX tiers are mixed-precision and use APEX names, so the widget cannot classify them (this also affects other APEX repos). Use the table below for sizes; mmproj and MTP files are never classified by the widget by design.

Files

Sizes are in GB (2³⁰ bytes), as displayed by the Hub. Full-GPU offload needs the file size plus the mmproj (if you use vision) plus the MTP head (if you use speculative decoding) plus KV cache and overhead.

I- files are quantized with an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces — no Wikipedia). In APEX evaluations, I-variants trade a negligible perplexity increase for better downstream accuracy and lower KL divergence. The Mini, Micro and Nano tiers use the diverse imatrix by design (per the APEX method), which is why they ship as I--named files (I-Micro, I-Nano) with no separate non-imatrix version.

The model

Qwen3.8-35B-A3B-Distill is Empero's distillation of the Qwen3.8 frontier teacher models (Qwen3.8 2.4T A95B and Qwen3.8 Flash Next) into the Qwen3.6-35B-A3B Mixture-of-Experts architecture — off-policy SFT on curated teacher chain-of-thought traces spanning math, code, general reasoning, instruction following, and tool use. The goal: bring frontier-scale reasoning behavior into a sparse 35B model that activates only 3B parameters per token.

• ~35B parameters, Mixture-of-Experts: 256 routed experts, 8 active per token, plus a shared expert.
• 40 layers, hybrid attention: ten cycles of three Gated DeltaNet (linear-attention) layers followed by one Gated Attention layer (qwen3_5_moe architecture, same family as Qwen3.6). Only 10 of 40 layers use full attention, so KV-cache overhead is minimal.
• Multimodal base, text-only fine-tune: the vision tower from the Qwen3.6-35B-A3B base is retained in the checkpoint but was not touched or evaluated by this distillation; 262,144-token native context.
• Reasoning: always-on distilled chain-of-thought — every response opens with a <think> block learned directly from Qwen3.8 teacher traces, rather than an adaptive effort toggle.
• Tool calling: native Qwen3.6 function-calling spec, no wrapper or tool-specific fine-tune required.
• Multi-Token Prediction: trained with an MTP head for self-speculative decoding, shipped here as separate MTP GGUFs.
• License: Apache-2.0.

How APEX quantizes Qwen3.8-35B-A3B-Distill

Routed experts are ~94% of the weights here, but only 8 of 256 fire for any given token, so they tolerate lower precision than the parts every token passes through. Attention + SSM are only ~2.7% of the weights but dense, and the shared expert (~0.4%) is always active and heavy-tailed. APEX classifies each tensor by role and applies a layer-wise precision gradient — edge layers keep higher precision, middle layers compress harder:

Embeddings and the LM head (~3% of weights) are kept at Q6_K (Q3_K on Mini) to preserve output-logit fidelity. Routers and norms stay F32. Because routing decisions use full-precision gates, quantization noise in inactive experts never affects the output.

For the tier configs and reproduction scripts, see the APEX repository.

Usage

Needs a recent llama.cpp build with qwen35moe (Qwen3.5 / Qwen3.6 A3B family) support. The low tiers additionally use IQ1_M / IQ2_XS / IQ2_S / IQ2_XXS expert formats, and the BF16 mmproj/MTP files need current BF16 support — when in doubt, use the latest release.

# text
llama-cli -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf -p "Your prompt" -ngl 99

# OpenAI-compatible server (tool calls + reasoning via --jinja)
llama-server -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --mmproj mmproj-BF16.gguf \
  --jinja --host 0.0.0.0 --port 8080 -ngl 99

# vision, standalone
llama-mtmd-cli -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --mmproj mmproj-Q8_0.gguf \
  --image image.png -p "What is in this image?" -ngl 99

# self-speculative decoding with the MTP draft head
llama-server -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --model-draft mtp-Qwen3.8-35B-A3B-Distill-Q8_0.gguf \
  --jinja --host 0.0.0.0 --port 8080 -ngl 99

The draft-model flag name/support for MTP-based self-speculative decoding varies by llama.cpp version (--model-draft / -md) — check your build's --help if it's rejected.

The GGUF embeds the official chat template, including the always-on thinking format and the native Qwen3.6 function-calling spec. Every response opens with a <think>...</think> block; parse and strip it for end users.

Recommended sampling parameters (from the empero-ai model card):

• temperature: 0.6
• top_p: 0.95
• top_k: 20
• Allow generous output length — 16,384 tokens recommended, since the model's chain-of-thought precedes the final answer.

Choosing an mmproj

mmproj-BF16.gguf (0.88 GB) is the full-precision projector; mmproj-Q8_0.gguf (0.60 GB) is the quantized one. Either pairs with any APEX tier above. If you are not using images or video, skip the mmproj entirely.

Choosing an MTP head

Three MTP draft heads are provided for self-speculative decoding: mtp-Qwen3.8-35B-A3B-Distill-BF16.gguf (3.74 GB, full precision), mtp-Qwen3.8-35B-A3B-Distill-Q8_0.gguf (1.99 GB, recommended balance), and mtp-Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf (1.26 GB, smallest / fastest to load). Any pairs with any APEX tier above. If you are not using speculative decoding, skip the MTP file entirely.

Notes

• MTP head: unlike some quants in this family, the Qwen3.6-35B-A3B base (and this distillation) was trained with Multi-Token Prediction, so a working MTP head is shipped here as separate GGUFs rather than baked into the main file.
• Benchmarks: no perplexity or accuracy benchmarks were run on these specific files. The published APEX numbers are from the reference Qwen3.5-35B-A3B evaluation in the APEX repository; see the base model card for Qwen3.8-35B-A3B-Distill's own benchmark results against the Qwen3.6-35B-A3B base.
• License: Apache-2.0, same as the base model.

Credits

• Base model: empero-ai/Qwen3.8-35B-A3B-Distill by Empero (Apache-2.0), distilled from Qwen3.8 teachers onto Qwen/Qwen3.6-35B-A3B (Alibaba Qwen team)
• Quantization method: APEX by the LocalAI team — built on llama.cpp
• Quants: MrFuzzihead