GLM-5.3-Flash — EXL3 4.65bpw
EXL3 quantization of GLM-5.3-Flash.
Quantized from zai-org/GLM-5.3-Flash-BF16
(the 642 GB BF16 master), not the default FP8 checkpoint — so the trellis is fitted to the
original weights rather than to already-rounded ones.
Requires ExLlamaV3 v1.4.5 or newer.
Details
| Size on disk | 189.6 GB / 176.6 GiB, 35 shards |
| Effective bitrate | 4.67 bpw (requested 4.65 with -hq), head 6 bpw |
| Conversion | exllamav3 v1.4.5, -b 4.65 -hq, default 250×2048 calibration |
| MTP | included, inline as layers.45 — no separate mtp.safetensors |
| Vision tower | included |
Per-layer reconstruction over the 45 decoder layers: mean rfn 0.011789, mean sqnr 46.19 dB.
-hq allocates unevenly — layers 0-14 and 30-44 land at ~5.04-5.68 bpw while layers 15-29 sit at
~4.07-4.09, so read per-layer error against its own bitrate rather than a single threshold.
Running it
Needs roughly 210 GiB of VRAM for weights plus a large cache.
Measured on 4× CMP 170HX (64 GB each, sm_80) under TabbyAPI:
| metric | value |
|---|---|
| prefill | 924 tok/s @ 132,045-token prompt |
| decode | 29.7 tok/s |
| context | 262,144 (cache_size 1,048,576) |
| VRAM used | 214,796 MiB / 209.8 GiB |
| MTP acceptance | 61-74% |
Two notes if you serve it with TabbyAPI:
- MTP is inline.
draft_mode: mtpfinds it with nodraft_model_name; the log should sayUsing main model MTP component for drafting. A benign!! Replaced 6 tensors/Overriding ...layers.45.*block at load is expected (dedup across the last two shards). - Use an explicit
gpu_split, not autosplit. Autosplit fills cards sequentially and packed three of four to ~99%, leaving no room for the KDA prefill workspace: 625 tok/s prefill with 996 MiB free at peak.gpu_split_auto: falsewithgpu_split: [54, 54, 54, 54]gave 924 tok/s and 5.6 GB free — same footprint, 48% faster.
Credits
Model by Z.ai. Quantization format and tooling by turboderp. Licensed MIT, as upstream.