GLM-5.3 · exlr8 (K2/K3/K4)

Version 0.5 (git tag v0.5). See the changelog. Weight set ID: 2011c0cedde962d19fc43e74c1d409279e398d40d27a6fdb4bcb7b49ea2ea6fb (see Identifying the weights).

Status: draft / preview. End-to-end gates (TP agreement, held-out KL against BF16, decode speed) are running. Metrics on this card will be updated as they land.

These weights cannot be used without E4M3 rounding. A loader must round every value the trellis codebook produces to FP8 E4M3 (round to nearest even, saturating) before applying the side vectors. The encoder searched only over E4M3-rounded values, so the rounding is part of the format: a stock EXL3 decoder, or any decoder that skips the rounding, reads measurably worse weights from the same bytes.

A quantization of GLM-5.3 (744B MoE: 256 routed experts × 75 MoE layers plus the MTP layer 78, dense layers 0-2) in the exlr8 v2 format, built for NVIDIA GB10-class unified-memory devices served at tensor-parallel sizes 3-7 (TP=4, 5 and 6 are the main targets; see "Known results so far" for what has been tested).

How does this differ from other quantization formats?

EXLR8 is a new quantization format. It reuses EXL3's trellis encoding — QTIP-style codes where each 16×16 tile is a path through a 16-bit-state trellis, every weight contributing K new bits, found by Viterbi search over the mcg codebook — but re-engineers everything around it for MoE models served on GB10-class hardware (128 GB unified LPDDR5X, ~220 GB/s streaming), where decode speed is bound by how many bytes of expert weights you read per step. A K-bit tile is 256×K bits, packed exactly as exllamav3's pack_trellis packs it (16K little-endian int16 in tensor-core order) and decoded by exllamav3's reconstruct plus the rounding below.

On top of the stock codec, EXLR8 changes four things:

1. FP8-rounded codebook. Codebook values are rounded to E4M3 inside the Viterbi search, so decoded weights are exactly FP8 and prefill runs on FP8 tensor cores. Rounding after the search instead would cost +4.0% (K3) / +15.3% (K4) error; a stock EXL3 decoder reading the same files gets +2.3%.

2. Deployment-oriented storage. EXL3 quantizes a whole tensor at one width. EXLR8 stores every routed expert at all three widths at once, and the width is chosen per expert-half at load time from a manifest — no re-encode to change the mix. Storage granularity is a 128-channel half (paired into 256-channel fragments), so a knapsack over per-half error tables can buy quality where it matters and cut bits where it doesn't.

3. One activation rotation per fan-out point. Each weight is decoded as w = R(diag(suh)·L(T))·diag(svh) — sign, scale, and a fixed 128-block Hadamard on each side (the Hadamard is inherited from exllamav3). The difference from per-tensor EXL3 is what's shared: on the hidden side, one sign vector per layer (signs.in/signs.out) and block-constant scales are used by every expert and every width, so the kernel rotates the token's activation once and shares a single A operand across all active experts instead of once per expert. Per-expert signs live on the intermediate side, folded into the stored weights. Against per-expert signs, sharing costs nothing in quality (NMSE ratio 0.9996–1.0003) and cuts MoE prefill at layer 40 from 23.07 to 18.67 ms. Rounding is calibrated (LDLQ against Hessians from real activations): held-out layer error is 14% lower than with an identity Hessian, and round-to-nearest is 2.3–33× worse.

4. MoE/TP-native layout. The sharding rule (16e + h + L) mod N covers any TP=3–7 rank count with zero re-layout; kernels index experts through pointer tables rather than contiguous ranges; each weight exists once (removing the ~3.5 GiB/rank duplicate dense+decode copies); and 64 KiB-aligned safetensors make remote halves HTTP range-fetchable (~75 GB per node at TP=4 under the home manifest, instead of the whole checkpoint).

Why three widths?

Decode above concurrency 1 is bound by the bytes of expert weights read per step. Every routed expert is therefore stored at three trellis widths — K2, K3 and K4 (the shared expert at K3 and K4; layer 78's routed and shared experts at K3 only) — and the serving width is chosen at load time from a manifest, not at encode time. Codes do not nest between widths: each width is an independent, Hessian-aware trellis encode with an E4M3 codebook, so prefill runs on FP8 tensor cores with no quality cost.

What is in this repository

Layer 78 (the MTP layer) stores its routed and shared experts at K3 only, so every manifest uses K3 there.

A loader reads the layer's index file and the manifest, takes one width per unit (gate/up per 128-channel half, down per expert), and fetches only the halves its TP rank owns — so one set of files serves TP=3 to 7 with no re-encode and no per-rank copies. Loader gives the byte layout.

Manifests

The three manifests above ship with the files:

These are the routed-expert weights of layers 3-77 that each manifest loads: trellis tiles plus side vectors. The MTP layer 78 (K3 in every manifest) adds 3.6 GB to each. Shared experts and the dense and vocabulary files are not included.

Width selection is deliberately a deployment decision: expert routing is workload-dependent (hot sets of different workloads barely overlap), so deployments with a routing histogram can write their own manifest to trade memory and bytes-per-step against measured per-unit error. Error tables in each index file carry the measured held-out error per (gate/up half, down) at every width in one comparable unit.

Width profiles

A width profile is a manifest: it says which width each routed expert loads at. Every profile reads the same files, so switching profiles is a load-time choice, with no new download beyond the widths it uses.

These KL numbers come from our own full-vocabulary test, which is tougher than the short panels most GLM-5.3 quantizations report: it scores long held-out documents (base64, hashes and IDs included) and wikitext. On the four-window panel that other GLM-5.3 cards use, the same profiles score far lower (bit325: 0.025 mean). The numbers are only useful for comparing quantizations measured on this same test.

How to read it:

• Model size is everything a full copy of the profile loads: all routed and shared experts at the profile's widths, plus the 10.7 GB of dense and vocabulary files. Expert weights per box is the busiest box's share of the experts at that tensor-parallel size, before the KV cache.
• KL is the full-vocabulary KL divergence of each next-token distribution from the BF16 model's, averaged over every position of teacher-forced text, on held-out text (64 rows of 2,048 tokens from documents the calibration never saw: 131,072 positions) and on wikitext-2 (65,536 positions). Lower is better. The median shows the typical token, the mean the tail.
• The means look high next to dense models' numbers, and that is the model, not the format. GLM-5.3's top-8 routing turns small numeric differences into different experts: two BF16 runs that differ only in their batching score mean 0.11 (median 0.004) against each other on 32,768 positions of the held-out text. Compare profiles with each other, and compare other quantizations only on the same harness: an EXL3 3.25 bpw quantization with home's per-expert split (davidsyoung/GLM-5.3-EXL3-TR3-3.25bpw) scores 0.161 on the held-out text here.

Which to use:

• The bit profiles start from every routed expert at one width and step whole layers up in the order that lowered KL the most per byte: down projections before gate and up, K2 to K3 before K3 to K4, and within each step the layers where a bit matters most. bit225 is down1's size with 5-10% lower KL. bit325 is home's size with lower KL (1.5% on held-out text, 7% on wikitext).
• Speed: at the same size the bit profiles decode slightly slower. In gr8 on four GB10s, bit325 decodes about 2.5% slower than home (three boots each), and bit225 about 2% slower than down1 on prose (two boots each). Prefill is unchanged.
• Memory: with gr8's default KV pool, bit243 leaves about 7 GiB free per box at TP=3 and bit260 about 3 GiB; at TP=4, bit338 leaves about 3.5 GiB. home, bit325, bit338 and up1 do not fit at TP=3.

Two more profiles were scored to fit the allocation and are not shipped: every routed expert at K2 (held-out 0.266, wikitext 0.368) and every routed expert at K3 (0.156 and 0.120).

Identifying the weights

The weight set ID names one exact set of weight files: the sha256 of the lines <path> <size> <sha256>, one per weight file (experts/*.safetensors and dense/*.safetensors, sorted by path, each line ending in a newline). This release's ID is 2011c0cedde962d19fc43e74c1d409279e398d40d27a6fdb4bcb7b49ea2ea6fb. Every manifest and every index slice carries it as set_id, so a copy of the files can be checked against a manifest or slice without reading the weights: recompute the ID from hashes/ (or from the files), and compare. Manifests are not part of the ID, since they choose widths and change more often than the weights.

Index slices

index-slices/tp<N>/k<K>/rank<r>.safetensors holds, for TP size N (3-7), width K (2-4) and rank r, every half that rank owns at that width under the sharding rule (16 * expert + half + layer) mod N, so a rank can fetch and load its halves without the layer index files. Per layer LLL (layer 78: K3 only):

The metadata names the TP size, rank, width, layers, half_bytes, the rule, the set_id the slice was cut from and the sha256 of each source index file. In every file the offsets also follow 65536 + ((p * 256 + e) * 16 + h) * half_bytes (p = 0 gate, 1 up, 2 down), with the shared expert's halves after the routed ones.

Loader

The byte-level format, enough to write a loader. A pure-PyTorch reference loader is in kindlingai/exlr8.

Files are little-endian safetensors. Skip tensors named __pad.*. A matrix is handled as W (in, out), so rows are the input side and the HF tensor *.weight is Wᵀ.

Tile decode. A 16×16 tile at width K is 32K bytes (16K int16 words).

Reconstruction. W = diag(suh) · Hb · T · Hb · diag(svh), in fp32, with T the decoded tiles (in, out) and Hb block-diagonal with 128×128 blocks H/√128, H the Sylvester Hadamard matrix (H₁ = [1], H₂ₙ = [[H, H], [H, −H]]). In exllamav3 terms: preapply_had_r(preapply_had_l(T, 128) * suh[:, None], 128) * svh[None, :].

Experts. In experts/exlr8-layer-LLL.index.safetensors, kK.index[e, p, h] is the absolute byte offset and the length of half h (0..15) of projection p (0 gate, 1 up, 2 down) of expert e in experts/exlr8-layer-LLL.kK.safetensors. The manifest's layers[L] gives each width: gate[e][h], up[e][h], down[e]. The shared expert uses the tensors prefixed shared. with e = 0, at the widths in shared (gate[h], up[h], down). Each half decodes on its own, with its width's side data:

Here p is gate or up, repeat₁₂₈ repeats each of the 48 block scales over 128 channels, and tile rows and columns count from the start of the half.

Dense and vocabulary. dense/exlr8-dense-LLL.safetensors holds layer L's other weights, named without model.layers.L.. dense/exlr8-vocab.safetensors holds embed_tokens, lm_head and norm.weight (HF model.embed_tokens.weight, lm_head.weight, model.norm.weight). Norms and biases are stored as shipped under their names. A matrix M (HF M.weight) has M.record, int64 [8] = (format, out, in, K, side: 0 replicated, 1 output, 2 input, channels per shard, shared input rotation, solve method), and data tensors by format. Trellis decodes W (in, out) as above. The other formats give the HF weight (out, in) directly.

E4M3 is OCP FP8 E4M3 (float8_e4m3fn). The codebook rounds to it to nearest even, saturating.

Checksums (optional) are xxhash64 with seed 0:

• Expert half, kK.checksum[e, p, h]: its trellis bytes, then its 128 per-channel fp16 values (the svh or suh slice above), then the 48 fp16 block scales.
• Dense, M.checksum[b], per block b of 128 channels on the sharded side (128 output rows when replicated): for trellis, entry b, then the block's 128 values of the sharded side's vector (svh, or suh on side 2), then the whole other vector. For int4 on side 2, q4[b] then s4[b]. For FP8 block, rows 128b.. of fp8, then scale row b. Otherwise, rows 128b.. of each data tensor in the order listed.

Calibration

Encoded with Hessian-aware (LDLQ) rounding against a fixed ~2M-token calibration set (2048-token rows; 977 calibration rows and 64 held-out rows from whole unseen documents). The calibration text, activations and Hessians are not part of this repository. Quality is scored against the BF16 model on the held-out rows.

Loading

kindlingai/gr8 serves this release on GB10 boxes at TP=3 and 4, and fetches each box's part of it from this repository. A pure-PyTorch reference loader, for checking a loader or kernel against known-good weights, is in kindlingai/exlr8. This repository holds everything a loader needs: the quantized weights, the manifests, and the base model's config.json, tokenizer and chat template.

The format's three exllamav3-derived primitives (the mcg trellis codebook, the 128×128 Hadamard matrix, and the tile packing order) are taken from exllamav3 (MIT) unchanged.

Known results so far

• Encoder output is bit-identical across boxes, page sizes and images (deterministic distributed encode).
• A pilot layer spliced into the previous-generation checkpoint decodes and prefills within noise of it.
• Full-vocabulary KL against BF16 for every shipped profile: see Width profiles.
• In gr8 with its default KV pool, home, bit325 and bit338 serve at TP=4, and down1, bit225, bit243 and bit260 at TP=3: smoke tests pass and an 87K-token needle is found. Speed and memory are in Width profiles.

Intended use and limitations

• Serving GLM-5.3 on GB10-class devices at TP=3-7. TP=5 needs engine support that is still landing.
• The DFlash2 drafter is not distributed with this checkpoint (its upstream licence does not permit redistribution); a deployment that wants it converts it at load time from the original files.
• Quantized weights: measure quality for your workload before relying on the lower-width manifests; down1 trades measurably more error for ~30% fewer bytes per step.
• Not for training. Router, norms and small projections are kept at their shipped precision (BF16/FP32) because routing is sensitive.

Licence

These weights are a quantization of Z.ai's GLM-5.3 and remain subject to the GLM-5.3 License, whose full text is in LICENSE. Its conditions apply to this quantization as a derivative work.

The parts this quantization adds (the exlr8 v2 packing, manifests and error tables) are also offered under CC BY 4.0: redistribution and modification with attribution are permitted. CC BY 4.0 does not remove or relax any condition of the GLM-5.3 License.

The exlr8 v2 format and the loader are separate works; the three exllamav3-derived primitives remain under their MIT licence.

Acknowledgements

exllamav3 made this possible through their original research on the EXL3 format.

Built on Z.ai's GLM-5.3; the encoder and the format's primitives are derived from exllamav3 0.0.43 (MIT).

brandonmusic's EXL3 quantizations of other models were an inspiration for some of this work.

Changelog

Each version is a git tag on this repository. The expert, dense and index files are unchanged since 0.1, so every version has the same weight set ID.

0.5 (2026-10-11)

Changes:

• Adds the bit225, bit243, bit260, bit325 and bit338 width profiles, with their sizes and full-vocabulary KL against BF16 in Width profiles. The weights and the set ID are unchanged.
• The card says where to load the release, now that gr8 is public.

0.4 (2026-10-10)

Changes:

• Adds the base model's config.json, generation_config.json, tokenizer and chat template, unchanged from zai-org/GLM-5.3-BF16 (9d2398f), so the repository is a complete model. The weights, widths and set ID are unchanged.
• Adds the byte-level Loader section and points to the reference loader.

0.3 (2026-10-08)

Changes:

• Adds the weight set ID (set_id) to the card and every manifest. The weights and widths are unchanged.
• Adds index slices for TP 3-7 at K2, K3 and K4, with their hashes.

0.2 (2026-10-08)

Fixes:

• home and up1 asked for K4 on layer 78's shared expert, which is stored at K3 only, so loaders failed on it. Both now use K3 there.
• Removed exlr8-manifest-calib-325.json. Its widths came from calibration routing counts, which the format spec rules out for shipped manifests.
• LICENSE now carries the full GLM-5.3 License text, which the licence requires in derivative works. The weights remain under it, and CC BY 4.0 covers only what this quantization adds.

Changes:

• The card states that loaders must apply E4M3 rounding.
• Manifests gain version and changelog fields, name layer 78's shared expert in mtp_layer, and no longer name an internal mount path in source.
• Card corrections: file paths, the layer 78 widths, the baseline for the −14% layer error (an identity Hessian, not round-to-nearest), an unsourced 181 GiB size, the loader (not public yet) and where to get config.json and the tokenizer.

0.1 (2026-10-08)

Initial release: routed experts of layers 3-77 at K2, K3 and K4 and of the MTP layer 78 at K3; dense and vocabulary files; home, down1, up1 and calib-325 manifests.