Hy4-preview GGUF
GGUF quantizations of tencent/Hy4-preview, made with llama.cpp.
Chinese version: README_zh.md
780B total parameters, 78 layers. DeepSeek-style MLA attention (q-LoRA rank 2048, kv-LoRA rank 512) with a gated output path and learnable attention sinks, wrapped in iHC hyper-connections on every attention and MLP block. 256 routed MoE experts with top-8 routing plus one shared expert. DSA sparse attention: 21 of the 78 layers own a full indexer, the rest share the preceding full layer's top-k. Vocabulary 120,832.
Quantized from the official BF16 weights. Every tier is imatrix-calibrated and ships as 39 shards.
Quantizations
| Tier | Size | Shards | BPW | PPL (wikitext-2) | Mean KLD | Same top-1 |
|---|---|---|---|---|---|---|
| master (BF16, not in this repo) | 1435.7 GiB | 39 | 16.02 | 2.9361 ± 0.0253 | — | — |
IQ4_XS | 388.3 GiB | 39 | 4.33 | 3.0789 ± 0.0157 | 0.0793 ± 0.0008 | 90.50 % |
Q2_K | 269.6 GiB | 39 | 3.01 | 3.8898 ± 0.0217 | 0.3451 ± 0.0030 | 80.35 % |
IQ2_XS | 218.9 GiB | 39 | 2.44 | 4.9648 ± 0.0294 | 0.5988 ± 0.0044 | 73.44 % |
The master row is not a file in this repo. It is listed so the numbers above have a reference point — the 1.4 TiB BF16 master was actually loaded and measured on the same machine, so every tier is compared against a real baseline rather than an estimate.
Pick IQ4_XS unless you cannot fit it. It keeps the same top-1 token as the BF16
master on 90 % of positions and its KL divergence is 4.4× lower than the next tier down.
Q2_K is the balanced choice and the best option for CPU-only inference. IQ2_XS exists
for when the footprint is hard-capped.
What the imatrix is worth
Every tier here is calibrated with an importance matrix. To quantify what that is worth,
Q2_K was built twice — once with the imatrix and once without — at byte-identical
size (276,118 MiB both times), so the only variable is the calibration:
| PPL | Mean KLD | Same top-1 | RMS Δp | |
|---|---|---|---|---|
Q2_K without imatrix | 4.2988 ± 0.0238 | 0.4543 ± 0.0034 | 76.98 % | 24.82 % |
Q2_K as shipped | 3.8898 ± 0.0217 | 0.3451 ± 0.0030 | 80.35 % | 21.36 % |
| improvement | −9.5 % | −24.0 % | +3.4 pts | −13.9 % |
Measured against this repo's own size-quality curve, that 24 % KL reduction is worth
+0.257 BPW — i.e. reaching the same quality without an imatrix would need a file
8.5 % larger (about 23 GiB more on the Q2_K tier). The imatrix itself costs nothing
in the shipped weights.
Where the gain lands, by KL percentile:
| Percentile | without | with | gain |
|---|---|---|---|
| 10 % | 0.00155 | 0.00058 | −62.8 % |
| median | 0.16232 | 0.09597 | −40.9 % |
| 90 % | 1.2245 | 0.9229 | −24.6 % |
| 99 % | 3.7199 | 3.3953 | −8.7 % |
| 99.9 % | 6.5440 | 6.4235 | −1.8 % |
The improvement is concentrated in the body of the distribution, not the tail: an imatrix makes the typical token markedly closer to the original model, while the rare catastrophic divergences are a property of low-bit quantization that calibration cannot undo. This is exactly what per-channel importance weighting is expected to do.
Calibration corpus: 476 KiB of real-world text, 220 chunks at n_ctx=512, run over the
full BF16 master (6.7 h on 8× H100).
What is protected
The routed experts are 96.5 % of this model's weights, so everything else can be kept at high precision for almost no size cost. It is kept:
| Tensors | Type | Reason |
|---|---|---|
ffn_gate_inp / exp_probs_b | F32 | MoE router; compressing it routes tokens to the wrong experts |
ffn_*_shexp | Q8_0 | the shared expert runs on every token |
hc_attn_* / hc_ffn_* / output_hc_* | F32 | iHC hyper-connection streams, every layer |
indexer.k_norm / indexer.proj | F32 | DSA top-k selection; a wrong pick discards the whole attention span |
indexer.attn_k / indexer.attn_q_b | Q8_0 | DSA indexer projections |
attn_kv_a_mqa / attn_k_b / attn_v_b / attn_q_a | Q8_0 | MLA low-rank paths are already narrow; compressing them damages the KV cache |
attn_gate / attn_sinks | Q8_0 / F32 | gated-MLA output path and learnable sinks |
ffn_gate / ffn_up / ffn_down | Q6_K | the single dense MLP layer before the MoE stack |
token_embd / output | Q6_K | a global type would otherwise squeeze these hard |
Usage
# Point at the first shard; llama.cpp finds the rest on its own.
llama-cli -m Hy4-preview-IQ4_XS-00001-of-00039.gguf
Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself, and an
explicit -ngl — including 0 — aborts that fitting and can fail to allocate compute
buffers even when the weights would have fit.
Every tier ships as 39 shards. Download all 39 into one directory — you only ever name
-00001-of-00039 on the command line.
Requires a llama.cpp build with hy_v4 support (merged upstream 2026-09-04, PR #28127).
Earlier builds will not load these files.
About the numbers
wikitext-2 test, n_ctx=512, 655 chunks (~335 K tokens), every tier through the exact
same command on 8× H100 80GB. KL divergence and top-1 agreement are computed against
logits saved from the BF16 master over 200 chunks.
These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.
Two notes on why the error bars here are small enough to be useful: the PPL runs use 655 chunks rather than the handful common in quantization write-ups, and KL divergence is reported alongside PPL because perplexity is a mean statistic that hides where a quantized model actually breaks. The percentile table above is only readable because both were done.
License
Apache-2.0, inherited from tencent/Hy4-preview — see LICENSE for terms. Quantized by qtum.