WarpQuant Qwen3.8-27B R16E4H4 Text
This is the text-only Qwen3.8-27B checkpoint produced with WarpQuant. It contains the causal language model, tokenizer, and chat template. The vision tower and MTP layer are not included.
Technical report · Code · Multimodal model
Format
WarpQuant applies a signed Hadamard rotation and 3-bit group quantization to 400 projection tensors. An Output-Fisher score selects a 0.05 bpw weak-column recovery budget in the original activation coordinates. The token embedding and output head use grouped INT4.
| Scope | Value |
|---|---|
| Text parameters | 26,895,998,464 |
| Analytical payload | 3.616506953 bpw |
| Analytical payload size | 11.323668 GiB |
| Calibration | 512 sequences × 512 tokens |
| Group / tile size | 128 / 128 |
The first release checkpoint stores reconstructed tensors in BF16 for direct Transformers evaluation. Its physical safetensor size is therefore larger than the analytical packed payload.
The text-only export was compared against the source conditional-generation checkpoint on the same tokenized prompt. The last-token logits matched exactly (max_abs_logit_error = 0, mean_abs_logit_error = 0, identical top-1), followed by a deterministic generation smoke test.
Evaluation
| Format | Text bpw | Payload | WT2 PPL ↓ | ARC-299 ↑ | MMLU-13,943 ↑ | Commonsense ↑ | GSM8K-500 flex ↑ |
|---|---|---|---|---|---|---|---|
| BF16 | 16.00 | 50.11 GiB | 6.9548 | 52.17 | 43.07 | 79.23 | 70.40 |
| Q4_K_M | 4.92 | 15.41 GiB | 6.9656 | 50.84 | 42.90 | 79.23 | 75.20 |
| IQ3_S | 3.6940 | 11.57 GiB | 7.1820 | 52.17 | 42.97 | 78.83 | 59.40 |
| WarpQuant R16E4H4 | 3.6165 | 11.32 GiB | 7.4737 | 56.86 | 42.72 | 78.83 | 61.00 |
Commonsense is the macro average of fixed 1,000-example HellaSwag, WinoGrande, and PIQA screens. GSM8K uses the same first 500 examples, 5-shot prompts, and flexible-extract accuracy for all four models.
Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4-Text"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
Citation
@misc{choi2026warpquant,
author = {Harim Choi},
title = {WarpQuant: Dual-Domain LLM Quantization via Hadamard Rotation and Output-Fisher Sensitivity},
year = {2026},
url = {https://github.com/HarimxChoi/WarpQuant}
}