GLM-5.3-Int4-Int8Mix

The first Int4/Int8Mix quantization of big GLM-5.3 (743B) — produced with the QuantTrio recipe, verified with fail-closed structural + numerical gates, and served at TP4 on four NVIDIA DGX Spark (GB10 / sm121) nodes over a RoCE fabric.

• Base: zai-org/GLM-5.3-BF16 (1507 GB, genuine BF16 — chosen over the fp8 repo so the router, DSA indexer, lm_head and layer 0 stay true BF16)
• Size: 377.4 GiB, 282 shards, compressed-tensors / pack-quantized
• Per-rank at TP4: 94.6 GiB
• Companion repo (scripts + raw results): https://github.com/chishiki37/glm-5.3-int4-int8mix-4x-dgx-spark

Serving (TP4 on 4× DGX Spark, NFS-served weights)

Status: validated 2026-09-09 · Nodes: 4 (gx10-141d head + 3 NFSoRDMA workers) · Context: 200K (1M in config, KV-pinned pool 200,064 tok)

Weights live in one copy on the head node (gx10-141d, 200G IB) and are NFSoRDMA-exported to the three workers — no per-node NVMe copies. This topology supersedes the all-local-NVMe lane (27.2/59.1/86.9 on 1d49+3b24+cb98+04af, 2026-08-30): it matches-or-beats it while keeping 3× ~405 GB of NVMe copies off the workers. Head reads weights locally (the NFS server never mounts its own export); workers mount the same path over RDMA (proto=rdma, vers=3, port 20049), so one uniform launcher runs on all ranks. Measured NFS: single-stream O_DIRECT 507 MB/s, 4-stream aggregate 3.6 GB/s; weight-load phase ≈ 7 min for 3 concurrent NFS ranks + 1 local rank.

Sweep-optimized winner config (autoresearch over spec k, max-seqs, NCCL channels, Marlin atomic-add, cudagraph mode, rail count — full table in the companion repo):

--speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_tensor_parallel_size":1,"attention_backend":"FLASHMLA_SPARSE"}'
--tensor-parallel-size 4 --nnodes 4 --master-port 29552 --max-model-len 200000
--max-num-seqs 16 --max-num-batched-tokens 8192
--gpu-memory-utilization 0.91 --kv-cache-memory-bytes 10950000000 --kv-cache-dtype fp8_ds_mla
--compilation-config '{"cudagraph_mode":"FULL"}' --enable-prefix-caching --async-scheduling
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice --trust-remote-code

The three winning knobs vs the local-weights lane: dual-rail NCCL (NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0, GID index omitted), Marlin atomic-add (VLLM_MARLIN_USE_ATOMIC_ADD=1), and max-num-seqs 16 (was 6/8). Doctrine unchanged: NCCL_MAX/MIN_NCHANNELS=4, HF_HUB_OFFLINE=1, pinned KV (never unpinned), swappiness=10, threshold-flushers only (drop iff MemAvailable<8 GiB). Boot ≈ 17–20 min.

Measured (e2e wall-clock, 256-token prose, temp 0, warm engine)

Winner = dual-rail + Marlin atomic-add + seqs16 (Σ 171.6). vs the all-local lane: C1 −3%/+1.6% (run-dependent), C4 −5%, C8 +3–7% — NFS parity at C1/C8 with 1.2 TB of worker NVMe freed. TTFT ≈ 0.6 s at ~780-token prompt · KV pool 200,064 tokens · MTP mean acceptance length 2.3–3.1 of 3.

Variance note: single-shot batteries swing ±10% on C1/C8 at identical config (24.94–27.64 C1). Marlin's C4 edge was the only consistently reproducible gain (3/3 runs ≥55.6). Run ≥2 batteries per arm before believing C1/C8 deltas under ~10%.

Integrity probes (NFS-lane health)

• Corruption probe: PASS — ~6K-char generations × 3 passes each in ko/en/tr, 0 bad tokens across ~54K chars.
• Long-context needle: PASS — 34,996-token prompt, all 3 keys (begin/middle/end) retrieved verbatim; prefill ≈550 tok/s, 63.7 s e2e.

Benchmarks (gx10:8000, single stream)

• GSM8K 98% (49/50, canonical set — 1 arithmetic slip)
• HumanEval-chat 92% pass@1 (46/50)

Quality (10 samples/task, temp 0, served via vLLM chat endpoint)

Ties-or-better on 6/7 vs the hosted GLM-5.2 reference (a cross-generation point, not the same model); the IFEval deficit is 2 questions at n=10 — noise-level, not a quant signal. Decode during the battery: 22–29 tok/s single-stream mixed content.

Quantization recipe

Config groups transferred verbatim from QuantTrio/GLM-5.2-Int4-Int8Mix — GLM-5.2 and GLM-5.3 are structurally identical (78 layers, hidden 6144, kv_lora_rank 512, 256 routed experts, first_k_dense_replace 3). Data-free RTN: no calibration, static, symmetric, weight-only.

Kept full precision: all of layer 0, every mlp.gate (MoE router), the DSA indexer/indexers_proj, MTP eh_proj/enorm/hnorm/shared_head, and lm_head.

Quantized by shard streaming (read one BF16 shard → quantize → pack → write one output shard; ~10 GiB peak RAM, resumable), using compressed-tensors' own calculate_qparams / quantize / pack_to_int32, with scales rounded to bf16 before quantizing so the persisted scale is exactly the one quantization assumed. Runtime: ~30–57 min CPU-only on one DGX Spark depending on source medium.

Verification (fail-closed gates)

• Structural: 282/282 shards open clean; exact 1:1 key mapping (59,585 source tensors → 58,992 quantized modules + 593 BF16/FP32 keeps); census 57,600/616/776 matches the pre-quant dry-run exactly; sacred modules plain BF16, zero packed leaks.
• Numerical: dequant vs BF16 on sampled tensors across all three groups — W4-g128 nrmse ≈ 0.111 (exactly theoretical for data-free absmax RTN), W8-g128 ≈ 0.0072, W8-channelwise MTP ≈ 0.009. Packing convention proven by discrimination (wrong unpack scores cos ≈ −0.58).
• Integrity: full 282-entry sha256 manifest (.sha256 per shard in this repo); distributed to 4 cluster nodes with per-target verification, 282/282 OK each.

Provenance

Quantized, verified, and benchmarked on a fleet of DGX Spark nodes over a RoCE fabric by an agent-assisted pipeline (Aug 29–30 2026; NFS serving lane validated Sep 9 2026). Companion repo with the verification gate script, launch recipe, autoresearch sweep orchestrator, quality battery, and all raw JSONs: https://github.com/chishiki37/glm-5.3-int4-int8mix-4x-dgx-spark

Note: the one-off streaming quantizer driver was ad-hoc and is not preserved; the verifier encodes the full recipe invariants (group census, sacred-module allowlist, packing convention) needed to reproduce or validate a re-quantization.