GLM-5.3-Flash EXL3 K2
Community EXL3 K2 pack of zai-org/GLM-5.3-Flash-BF16: 2-bit MCG trellis on routed MoE experts only. Attn / shared / embed / head / vision stay source-native. 120 safetensors shards, 97,728,721,536 B (91.017 GiB).
These weights need a specific vLLM build (K-pool fix shipped 2026-08-30; reinstall)
Stock vLLM cannot load this pack. It has neither the
exl3quantization method nor theGlm5Nextarchitecture, and no flag turns them on.pip install vllmwill appear to work and then fail after you have downloaded 91 GiB.Prebuilt wheels, no compiler, minutes to install: vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm
| Where | What |
|---|---|
| This repo | the weights, 120 shards, 91.017 GiB |
| spark-vllm | prebuilt vLLM + ExLlamaV3 wheels for GB10 |
| recipe | install scripts, serve flags, full measurements |
| Format | EXL3, bits=2, codebook mcg, scope glm53_routed_experts_only |
| Architecture | Glm5NextForConditionalGeneration |
| Spark (TP=1) | native MTP k=2: 15.7–16.5 tok/s @ 8k; 14.6–15.7 tok/s @ 64k |
| Max ctx allocated | 131072 (KV 786,432 @ util 0.91); recommended serving ctx 65536, see ladder |
| sixcat 0.5.1 | 120/120 think-on @ 64k — overall 84.17 on the fixed 2026-08-30 runtime (flagged: mmlu:8 and ifeval:1300 hit their budgets), same overall as the earlier container run |
| KLD vs BF16 (fidelity suite v1) | 512 contexts, 1,048,064 positions, full-vocab KL through the shared head — token-mean 0.3346 nats (95% CI 0.320–0.349), median 0.117, p99 3.33, top-1 agreement 0.788; the suite's FP8 anchor is 0.028 / 0.943; the K2/K3 mix scores 0.3121 / 0.795 on the same contexts. Method and scorer validation in the recipe's docs/KLD.md |
| long context | boots at 262,144 (KV pool 1,093,332 tokens, 4.17x full-256k concurrency); real-text needle ladder verified to 163,479 prompt tokens (needle recalled verbatim, prefill ~590 tok/s, decode 17–20). Known runtime bug: prompts between ~163k and ~180k wedge the engine — keep prompts ≤163k until the fix lands; ≤131,072 contexts unaffected. Details in the recipe README |
| Also measured | 4× L40 TP=4 text /v1 smoke (see below) |
This is a quantized weight pack. Pull it, then follow the GitHub recipe for vLLM flags. Weights are MIT (Z.AI).
Quick start
# 1. the runtime first, so a bad environment fails in seconds instead of hours
git clone https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
cd GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
python scripts/preflight.py
bash scripts/install_prebuilt.sh
# 2. then the 91 GiB
hf download vcruz305/GLM-5.3-Flash-EXL3-K2 \
--local-dir ~/models/GLM-5.3-Flash-EXL3-K2
--local-dir resumes. There is no --resume-download flag. Last path component must be the Hub basename.
One-Spark vLLM (winner flags). Full notes, ctx ladder, and scripts: GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe.
export EXL3_FUSED_MOE=1
vllm serve ~/models/GLM-5.3-Flash-EXL3-K2 \
--served-model-name GLM-5.3-Flash-EXL3 \
--quantization exl3 \
--tensor-parallel-size 1 \
--kv-cache-dtype fp8 \
--max-model-len 8192 \
--gpu-memory-utilization 0.87 \
--max-num-seqs 1 \
--max-num-batched-tokens 2048 \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
--skip-mm-profiling --limit-mm-per-prompt '{"image":4,"video":1}'
Load log must show fused exl3_moe at bits=2. Bits=2 loading is handled natively by the EXL3 plugin in the runtime wheels, so there is nothing to patch. Do not pass --moe-backend marlin. Native MTP; do not mix a DFlash sidecar on the same server.
Eval/serve at 64k: MAX_MODEL_LEN=65536 GPU_MEM_UTIL=0.91 (same MTP k=2). Spec A/B was ranked at 8k so MLA page size stayed in one band.
Spark measurements (2026-08-29)
Hardware: one NVIDIA GB10 (SM121), ~121 GiB unified. Engine: vLLM --quantization exl3, EXL3_FUSED_MOE=1. Bench: streamed /v1, thinking off, 128 gen, seqs=1.
Spec ranking @ 8k
| Config | Decode tok/s | Accept |
|---|---|---|
| no spec | 9.6–9.8 | — |
| DFlash sidecar k=7 | 11.5 | mean ~1.8 / 7 |
| DFlash sidecar k=3 (older runtime) | 12.8 | mean ~1.8 / 3 |
| DFlash k=3 BF16, Triton draft attn | 15.6 | 26–79% by workload |
| MTP k=1 | 14.8 | 76–80% |
| MTP k=2 | 15.7–16.5 | ~74/44%, mean ~2.2 |
Context ladder (KV from engine log; decode = same 128-token bench when run)
| max_model_len | spec | util | GPU KV | Decode |
|---|---|---|---|---|
| 8192 | none | 0.87 | 192k–208k | 9.6–9.8 |
| 8192 | DFlash k=7 | 0.87 | 15,281 | 11.5 |
| 16384 | DFlash | 0.91 | 45,095 | /v1 pong |
| 32768 | DFlash | 0.91 | 90,035 | allocated (attn block 7168) |
| 8192 | MTP k=2 | 0.87 | 104,857 | 15.7–16.5 |
| 65536 | MTP k=2 | 0.91 | 786,432 | 14.6–15.7 |
| 131072 | MTP k=2 | 0.91 | 786,432 | allocates; 81,920-token prompt OK, 98,304 faults |
131072 allocates (KV 786,432, 6.00x) and an 81,920-token prompt completes, but a 98,304-token prompt faults in the EXL3 fused-MoE path and kills the engine, so 65536 stays the recommended serving ctx. DFlash at 8k / util 0.87 cannot climb ctx until util 0.91 (draft KV collapse).
sixcat 0.5.1 think-on, vendor glm-5.x, 20/category, host-guarded HumanEval, 64k serve: knowledge 65 · math 100 · truth 85 · instruct 75 · code 90 · tools 90. overall 84.2 with trunc-in-think:instruct — not a clean overall.
What is in the shards
- Routed experts: EXL3 K2 (
suh,svh,trellis,(1,)int32mcg) ongate_proj/up_proj/down_proj - 37,152 expert weights + source-native non-expert tensors
model.safetensors.index.json, tokenizer, processor, generation config, chat templatequantization_config.json:quant_method=exl3,bits=2,codebook=mcg,scope=glm53_routed_experts_only
Also measured: 4× L40 TP4
Text /v1 smoke on 2026-08-28, 4× NVIDIA L40 (SM89), TP=4, 8k, fused exl3_moe, no BF16 expert reconstruct. /health 200; 17 * 6 → 102. Build context under runtime/l40-vllm/ in this repo (CUDA 8.9, dense MLA fallback on L40). That gate is L40 TP4 text @ 8k. One-Spark GB10 flags and ctx live in the GitHub recipe.
Links
- Recipe and scripts: https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
- Prebuilt runtime wheels: https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm
- BF16 source: https://huggingface.co/zai-org/GLM-5.3-Flash-BF16
- GLM-5.3-Flash: https://huggingface.co/zai-org/GLM-5.3-Flash
License: MIT (Z.AI), same as the BF16 source. Community quant; not an official Z.ai release.