GLM-5.3-Flash-Uncensored-NVFP4

NVFP4 weight quantization of the abliterated (refusal-removed) build of Z.ai's GLM-5.3-Flash — a 320B / 18B-active Mixture-of-Experts model (glm5_next, 288 routed experts top-8 + 1 shared) with hybrid linear + sparse attention, 4-wide Manifold-Constrained Hyper-Connections (mHC), a native vision + video tower, an MTP speculative head, and a 1M-token context. The routed-expert weights are quantized to NVFP4 (4-bit, NVIDIA FP4 E2M1, group-16 + FP8-E4M3 group scales + FP32 global scale) — that is 97% of the parameters — while everything else is kept BF16, cutting the checkpoint from 306 GiB (block-FP8) to ~191 GiB with no measurable loss outside the experts. Browse all models in the OrcaRouter Model Catalog.

⚠️ Disclaimer — read before use

This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). It will comply with harmful, unethical, offensive, or illegal requests the original GLM-5.3-Flash would refuse — it has no meaningful built-in guardrails. Released strictly for legitimate research: interpretability, AI-safety / refusal-mechanism study, red-teaming, and robustness evaluation. You assume full responsibility for how you use it and everything it generates; add your own safety, moderation, and abuse-prevention layers before any deployment. Use must comply with the MIT License inherited from the base model and all applicable law. The authors accept no liability for misuse, and its outputs do not reflect the views of the uploaders or of Z.ai / Zhipu AI.

Model details

What's quantized

• OrcaSAQ — Sensitivity-Aware Quantization. This split follows OrcaSAQ, our calibration-free, architecture-aware policy: quantize the quantization-tolerant bulk (the routed experts, 97% of params) and keep every sensitive component at its source precision (attention, shared/dense MLP, router, indexer, embeddings, lm_head, vision, and the F32 aux). NVFP4 is a single 4-bit tier, so here OrcaSAQ decides what to quantize rather than raising per-role bit-widths (see the MLX build for the multi-bit version of the same policy).
• Weight-only, data-free. Expert weights are derived directly from the source checkpoint (symmetric FP4, per-group absmax); activations are quantized dynamically at runtime — no static calibration corpus. The abliteration is baked into the weights, so refusal-removal is preserved.
• F32 aux kept F32. Some public NVFP4 builds downcast the hyper-connection / SSM / router-bias aux tensors to BF16; those tensors are precision-sensitive (bf16 error is amplified through the SSM recurrence and the mHC residual), so this build keeps them FP32 at ~zero size cost.
• MTP block included. The multi-token-prediction head (layer 45, ~7.4 B parameters) ships with the build, de-risked to BF16 rather than quantized, so the weights match the num_nextn_predict_layers the config declares. Earlier revisions of this repo omitted it; if you pulled before this note was added, re-fetch the index and the shard-0006*.safetensors files.
• KV cache is not quantized (BF16 at runtime).

Recipe note. This mirrors the target split of the reference build RedHatAI/GLM-5.3-Flash-NVFP4 — experts-only NVFP4 — and is byte-format-identical to it (same weight_packed / weight_scale / weight_global_scale layout), so it loads through the same vLLM path. It differs in two ways: it is built from the abliterated checkpoint, and it keeps the F32 aux tensors at F32.

Requirements

• A vLLM build with glm5_next (GLM-5.3-Flash) support. This is a brand-new architecture and is not in stock vLLM yet (tracked in vllm#53906). Use the official image vllm/vllm-openai:glm53-flash-x86_64-cu129 (or the ROCm/arm/cu130 variants), or a build from that PR. transformers>=5.16 provides the reference arch.
• GPU. Verified serving on 8×H100 (Hopper) via the official image — vLLM runs the compressed-tensors NVFP4 path on Hopper. A Blackwell GPU (B100/B200/GB200) additionally has native FP4 tensor cores for best throughput. Plan for ~191 GiB of weights plus KV cache (e.g. 8×H100/H200 with tensor parallelism).
• Hopper does not support FP8 KV cache for this model — omit --kv-cache-dtype fp8 there.

Usage — self-host with vLLM (OpenAI-compatible)

docker run -d --name glm53-nvfp4 --gpus all --ipc=host -p 8000:8000 \
  -v /path/to/GLM-5.3-Flash-Uncensored-NVFP4:/model:ro \
  -e VLLM_SSM_CONV_STATE_LAYOUT=DS -e VLLM_KV_CACHE_LAYOUT=HND \
  vllm/vllm-openai:glm53-flash-x86_64-cu129 \
  --model /model --served-model-name GLM-5.3-Flash-Uncensored-NVFP4 \
  --tensor-parallel-size 8 --max-model-len 8192 --enforce-eager --trust-remote-code

Thinking control

GLM-5.3-Flash has no enable_thinking toggle — its chat template always opens a <think> block. Control the budget with reasoning_effort (low / high / max; defaults to max), and pass clear_thinking: true for chat:

client.chat.completions.create(
    model="GLM-5.3-Flash-Uncensored-NVFP4",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"chat_template_kwargs": {"reasoning_effort": "low", "clear_thinking": True}},
)

Give generation enough budget to reach </think>, or replies get truncated inside the scratchpad.

Evaluation

Measured on this build vs its block-FP8 abliterated reference. All numbers are from these exact weights, not inherited from the base card.

Weight-space fidelity (NVFP4 experts vs FP8)

Sampled across all 42 quantized expert layers (gate + down), NVFP4 round-trip vs the source:

9.18% is the intrinsic floor of 4-bit FP4 for these narrow experts (moe_intermediate = 2048) and is uniform across layers — no per-layer degradation.

Perplexity / KLD / Top-1 vs the FP8 reference (wikitext-2)

Forward-pass measurement of the NVFP4 recipe against the FP8-dequant reference (24,564 predicted tokens):

KLD 0.073 and Top-1 91.7% place this squarely in the 4-bit tier (for comparison, a 3-bit build of a sibling model lands at KLD ~0.17 / Top-1 ~90%). A direct vLLM load of this build measured PPL within the same band (a real-kernel W4A4 forward reads a few points higher than the weight-only forward, as expected for dynamic activations).

Uncensoring retained after quantization

Held-out JailbreakBench test split (64 harmful + 64 benign), served with vLLM on this exact build, greedy, reasoning_effort=low, <think> stripped before a rule-based classifier:

Harmful refusal stays in the single-digit-to-teens band — down from ~90% on the base model — and benign over-refusal is zero. The few extra points over the FP8 abliteration are the expected residual: the refusal direction lives almost entirely in the routed-expert down_proj matrices, and 4-bit quantization of exactly those weights leaves a little more of it behind than FP8 does. The build answers harmful requests directly (82.8% compliance) while often wrapping the answer in a disclaimer (a caveat, not a refusal). Refusal is reduced, not removed — some content categories the direction does not reach still refuse (see the base abliteration card).

Method note. The refusal metric is a rule-based opening-phrase classifier (bilingual EN/ZH), indicative rather than an LLM-judge / publication-grade number. Evaluate rigorously for your own use case.

Fine-tuning & re-quantization

• Loads through any vLLM build with glm5_next + compressed-tensors NVFP4 support.
• Abliteration is a weight edit, not data-level unlearning: fine-tuning on refusal-heavy / safety data can partially re-introduce refusals; neutral / task data preserves the uncensored behaviour.
• For a higher-fidelity build, keep the expert down_proj at FP8 (mixed-precision, larger); for a smaller one, a 3-bit MLX build trades ~2× the KLD for ~20 GiB.

Bias, risks, and limitations

• Safety guardrails removed — will produce harmful, biased, or offensive content on request.
• Inherits any biases and limitations of the base GLM-5.3-Flash.
• 4-bit expert quantization adds a small quality trade-off vs the FP8 source (see Evaluation) and leaves a few extra points of residual refusal.
• Capability is expected to track the base within measurement noise; the numbers above are on sampled corpora, not a full harness run.

License

MIT, inherited from the base model zai-org/GLM-5.3-Flash. Abliteration and quantization do not change the underlying license obligations.