GLM-5.3 tiny architecture fixture (glm_moe_dsa)
A ~0.27M-parameter random-init checkpoint that reproduces the reduced GLM-5.3 MoE config
schema so loaders, quant planners, and CI jobs can exercise the new glm_moe_dsa
architecture without touching the 753B weights.
What this is
- Base architecture:
zai-org/GLM-5.3(released 2026-08-25),model_type: glm_moe_dsa,architectures: ["GlmMoeDsaForCausalLM"], MIT license. - What it is: a byte-reproducible random-init checkpoint plus a reduced config that keeps
the real field names and the dense-to-MoE layer schedule (
first_k_dense_replace,moe_layer_freq, routed experts + shared experts). - What it is not: not trained, not distilled, not a quality or benchmark claim, and not a quantization of anything.
- Why it is useful: the base is 753,329,940,480 parameters (Hub safetensors metadata), so it cannot be instantiated in a unit test, in CI, or on a laptop. This fixture lets you test config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and safetensors load paths in milliseconds.
Geometry (base vs fixture)
| Field | Base GLM-5.3 | This fixture |
|---|---|---|
| num_hidden_layers | 78 | 4 |
| hidden_size | 6144 (read partially from base config; verify) | 64 |
| num_attention_heads / num_key_value_heads | 64 / 64 | 4 / 4 |
| head_dim | 192 | 16 |
| n_routed_experts | 256 | 8 |
| num_experts_per_tok | 8 | 2 |
| n_shared_experts | 1 | 1 |
| n_group | 1 | 1 |
| first_k_dense_replace | 3 | 1 |
| moe_intermediate_size | 2048 | 32 |
| intermediate_size (dense) | not captured | 128 |
| num_nextn_predict_layers | 1 | 0 |
| dtype | bfloat16 | float32 |
| vocab_size | 154820-class | 256 |
Intentional omissions (documented, not silent)
- No DeepSeek-style sparse-attention (DSA) indexer tensors.
- No MTP / next-n-predictor head (
num_nextn_predict_layers: 0). - No
lm_headtensor; a loader must tie tomodel.embed_tokens.weightor supply its own head. - Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.
Verification actually performed (stdlib only, no torch in this environment)
- safetensors header parses: 113 tensors, 1,097,984 data bytes = 274,496 float32 parameters,
contiguous
data_offsets, header padded to 8-byte alignment. checksums.txtrecords the SHA-256 of every tensor blob.- Deterministic regeneration: SplitMix64 seed 20260901, Box-Muller normals, scale 0.02, float32 row-major, consumed in sorted-name order.
- Generator script is included as
build_fixture.pyso the folder can be rebuilt and diffed.
Not yet verified: loading under a specific transformers version (no torch/transformers in
the build environment), and whether GlmMoeDsaForCausalLM accepts this reduced geometry without
extra fields. Treat those as open until run against a real install.
How to use
Read the tensors with the standard library (no torch needed, matching how this was built):
import json, struct
with open("model.safetensors", "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
# header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n
Or with the safetensors package:
from safetensors.torch import load_file
tensors = load_file("model.safetensors") # {name: tensor}
To exercise a real loader, build a config from config.json (the glm_moe_dsa
model type; use AutoConfig.from_pretrained(..., trust_remote_code=True) where
needed) and feed these weights in. There is no lm_head tensor and the tokenizer
files are placeholders, so supply your own head/tokenizer.
License
MIT, unchanged from the base model per its Hub metadata. See LICENSE.
Citation
Zhipu AI / Z.ai, GLM-5.3, 2026.
Buy me a beer:
BTC bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p