Spark-X2.5-4B — NVFP4 W4A16 (GPTQ)
NVFP4 weight-only (W4A16) quantization of XHToken/Spark-X2.5-4B
— 4-bit float weights in 16-element blocks with FP8 block scales, activations in
bf16. lm_head and every self_attn.g_proj (the head-wise attention-output gate)
kept in bf16, matching XHToken's own INT8 exclusion set. Rounding is GPTQ
(Hessian-aware), not RTN — this matters a lot here (see the eval). Produced with
llm-compressor 0.13.
- 2.7 GiB on disk (bf16 base is 7.7 GiB — ~65 % smaller)
- The first NVFP4 / 4-bit
compressed-tensorsbuild of Spark-X2.5-4B - Serves on vLLM on Blackwell via the out-of-tree Spark2_5 plugin (below); on SM120 the weight-only path decodes through the Marlin kernel to a bf16 GEMM (no native FP4 compute), so the benefit here is footprint, not raw speed
- Leaves ~10 GiB of KV headroom on a 16 GB card — room for a large slice of the 1 M-token context window
Evaluation
lm-evaluation-harness, vLLM 0.26 backend (+ Spark2_5 plugin), greedy decoding,
median of 3 draws with the range (the harness is non-deterministic run-to-run
even at greedy — a single draw is not a reproducible score). HumanEval-instruct
pass@1 / create_test (n = 164); MBPP base 3-shot (n = 500). RTX 5070 Ti
(Blackwell, SM120).
| build | HumanEval-inst | MBPP (3-shot) | size | Δ HE / MBPP vs bf16 |
|---|---|---|---|---|
| bf16 base | 79.88 % (79.88–81.10) | 48.80 % (48.60–48.80) | 7.7 GiB | — |
| NVFP4-W4A16 GPTQ (this) | 75.61 % (73.78–76.22) | 46.80 % (46.40–46.80) | 2.7 GiB | −4.3 / −2.0 pp |
| NVFP4-W4A16 RTN | 71.34 % (70.73–71.95) | 33.20 % (32.60–33.80) | 2.7 GiB | −8.5 / −15.6 pp |
GPTQ rounding is doing real work on this model — plain RTN loses 8.5 pp on HumanEval and collapses on MBPP (−15.6 pp, 33 %). GPTQ recovers +4.3 pp / +13.6 pp, landing at −4.3 pp HumanEval and −2.0 pp MBPP (the MBPP Wilson intervals overlap the bf16 baseline). For a 4-bit build at 65 % smaller, that is a usable trade. If you want maximum quality retention, XHToken's own FP8 build is the safer choice.
Serving (vLLM + out-of-tree plugin)
Spark2_5ForCausalLM is not in the vLLM model registry. Install the general
plugin into the same environment as vLLM:
git clone https://github.com/XHToken/Spark-plugin
pip install -e ./Spark-plugin --no-deps
pip install -U "openai>=2.25.0"
Then serve:
vllm serve Ttimms/Spark-X2.5-4B-NVFP4 \
--trust-remote-code \
--enforce-eager \
--max-model-len 32768 \
--kv-cache-dtype fp8 \
--enable-auto-tool-choice --tool-call-parser spark25
On WSL, also set VLLM_USE_V2_MODEL_RUNNER=0 (the V2 runner needs UVA, which
WSL disables). --enforce-eager is the safe default on SM120; weight-only
NVFP4-W4A16 also runs with CUDA graphs.
Method & provenance
- Quantizer:
llm-compressor0.13,GPTQModifier(scheme="NVFP4A16", targets="Linear", dampening_frac=0.1), ignorelm_head+re:.*self_attn\.g_proj+ tied embeddings. Calibration: 512 samples oftheblackcat102/evol-codealpaca-v1(code, disjoint from the eval sets), 2048 tok. - Base:
XHToken/Spark-X2.5-4B(Spark2_5ForCausalLM, ~4.1 B, hybrid sliding/full attention, head-wise attention-output gate, 1 M context, Apache-2.0). - Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26 + the Spark2_5 plugin.
License
Apache-2.0, inherited from XHToken/Spark-X2.5-4B.