Qwen3.5-4B — GGUF Q2_K_XL refined with GSQ
GGUF Q2_K_XL checkpoint of Qwen/Qwen3.5-4B
in which the discrete grid assignments have been refined with GSQ
(Gumbel-Softmax Quantization), starting from the public Unsloth GGUF
initialization and projected back into the same K-Quant format.
The optimized file runs unchanged on llama.cpp / Ollama and is a
drop-in replacement for the corresponding standard GGUF checkpoint.
- Paper: GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv:2604.18556)
- Paper page on HF: https://huggingface.co/papers/2604.18556
- Code: https://github.com/IST-DASLab/GSQ
- Collection: https://huggingface.co/collections/ISTA-DASLab/gsq
Quantization details
- Base model:
Qwen/Qwen3.5-4B - Starting point: Unsloth GGUF
UD-Q2_K_XL - Released quantization:
Q2_K_XL - Format: GGUF K-Quant
- Pipeline: Unsloth GGUF init → GSQ Gumbel-Softmax refinement → re-pack into the same format
- Runtime:
llama.cpp,ollama,LM Studio, anything that consumes GGUF
Storage layout
This file is bit-for-bit standard GGUF Q2_K. GSQ only changes the values of the quantized weights by relearning the discrete grid assignments inside each K-Quant block.
As a result:
- The file size matches the corresponding upstream Q2_K_XL file for the same model.
- Any
llama.cpp/ollamabuild that loads a regular Qwen3.5-4B Q2_K_XL GGUF should load this file with zero changes. - The Hugging Face UI reports the GGUF block type, e.g.
Q2_K_XL, rather than per-tensor dtypes. This refers to the on-disk K-Quant encoding, not the precision of any optimizer state used during GSQ refinement.
Evaluation
We compare the GSQ-refined checkpoint against the Unsloth UD-Q2_K_XL
checkpoint used as initialization.
| Model | AIME 25 | GPQA Diamond | IFEval | GSM8K | MMLU-Pro |
|---|---|---|---|---|---|
Unsloth UD-Q2_K_XL | 26.67 | 56.06 | 76.14 | 79.61 | 66.78 |
GSQ-refined Q2_K_XL | 60.00 | 67.17 | 82.97 | 88.48 | 70.87 |
Evaluation notes
All evaluations are zeroshot. The Unsloth UD-Q2_K_XL model was evaluated with thinking disabled on all
tasks, because its thinking mode was found to be broken and produced worse
results.
For the GSQ-refined model:
- AIME 25 and GPQA Diamond were evaluated with thinking enabled.
- IFEval, GSM8K, and MMLU-Pro were evaluated with thinking disabled.
Token Efficiency
For MMLU-Pro, both models were evaluated with a maximum output budget of 32K tokens.
- The Unsloth model consumed 55.32M output tokens in total.
- The GSQ-refined model consumed 41.82M output tokens in total.
Usage with llama.cpp
hf download ISTA-DASLab/Qwen3.5-4B-GGUF-GSQ \
Qwen3.5-4B-Q2_K_XL.gguf --local-dir .
./llama-cli -m Qwen3.5-4B-Q2_K_XL.gguf -p "Hello"