HyperPrune-Qwen3.6-27B-2to4
Qwen/Qwen3.6-27B pruned to 2:4 semi-structured sparsity with HyperPrune (Sun & Sakuma, Learning Semi-Structured Sparsity for LLMs via Shared and Context-Aware Hypernetwork, ICLR 2026, OpenReview).
This is a reproduction run produced at Elastix as part of the BLADE
sparsity-method comparison. It is plain sparse bf16/fp16 safetensors and
loads with stock transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("elastix-ai/HyperPrune-Qwen3.6-27B-2to4")
t = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-27B")
What differs from the paper's own recipe
| paper / repo default | this checkpoint | |
|---|---|---|
| calibration corpus | allenai/c4 | DKYoon/SlimPajama-6B, validation (BLADE's corpus) |
| pruned modules | see below | see below |
Everything else — hypernet architecture, both training stages, all learning rates, step counts, temperature, prior, row selection — is HyperPrune's own shipped setting.
Configuration
{
"model": {
"name_or_path": "Qwen/Qwen3.6-27B",
"dtype": "bfloat16"
},
"data": {
"dataset_name": "elastiml:elastix-ai/elastiml-calib-qwen3.6-27b",
"num_samples": 128,
"seq_len": 2048,
"seed": 42
},
"hypernet": {
"type": "mlp",
"hidden_dim": 256,
"emb_dim": 64,
"use_layer_emb": false,
"use_comp_emb": false,
"use_hessian_diag": true
},
"training": {
"sup_steps": 12000,
"sup_lr": 0.001,
"ft_lr": 0.0003,
"ft_nsamples": 4,
"rows_per_step": 400,
"cascade_inner_steps": 300,
"ft_mode": "cascade",
"tau": 0.5,
"prior_source": "sparsegpt",
"wanda_residual_alpha": 2.0,
"compensated_propagation": true,
"use_weight_compensation": true,
"train_on_compensated": true,
"fixed_rows_count": 200,
"fixed_rows_pos": "first",
"dense_layers_list": []
},
"output": {
"save_dir": "/home/ubuntu/hyperprune_work/outputs/hp-qwen36_27b-2to4",
"wanda_dir": "/home/ubuntu/hyperprune_work/outputs/hp-qwen36_27b-2to4_ref",
"preserve_wanda_dir": false
}
}
Measured
| metric | value |
|---|---|
overall decoder sparsity (check_sparsity) | 0.5003 over the 17.1B pruned MLP weights (whole-model 0.3183: attention stays dense per BLADE's two_four_experts spec) |
WikiText-2 PPL (HyperPrune eval_ppl.py, seqlen 2048) | 10.205 |
| WikiText-2 word PPL (lm-eval-harness, BLADE's protocol) | 13.56 |
| training wall-clock | 48.0 min |
| peak GPU during cascade FT | 14.05 GB |
| GPU | 1 x NVIDIA RTX PRO 6000 Blackwell (97 GB), CUDA 13.0, torch 2.13.0+cu130 |
Two things to know before comparing this number to the paper
1. Every decoder layer is pruned here. HyperPrune's own shipped configs set
dense_layers_list: [0, 1], leaving 2 layers fully dense and yielding ~46.9 %
sparsity rather than 50 %. This checkpoint prunes every layer, matching BLADE's
two_four_experts spec, so it is a true 2:4 model in the modules BLADE prunes.
2. Only a few percent of this mask was chosen by the hypernet. The shipped
recipe sets fixed_rows_count: 200, so the hypernet decides the mask for the
first 200 output rows of each projection and every remaining row keeps the
SparseGPT prior's mask verbatim. This is HyperPrune's own default, kept here
deliberately because the brief was to change nothing but the calibration corpus.
The two perplexity rows are different quantities and are not comparable to each
other. The first is token-level PPL over concatenated WikiText-2 at seqlen 2048
(the Wanda/SparseGPT convention). The second is lm-evaluation-harness
word_perplexity at max_length=2048, which is BLADE's protocol — pinned
empirically by reproducing BLADE's dense LLaMA-2-7B value of 9.19 (measured
9.1915).
Calibration
ElastiML self-distilled, not SlimPajama: elastix-ai/elastiml-calib-qwen3.6-27b, which is
BLADE's corpus for this model. The set is consumed from its pre-tokenized input_ids
verbatim rather than by re-encoding the text column, matching ElastiCore's
_generate_from_token_field. Measured effect on Phase A mean prior reconstruction:
0.020752 with ElastiML vs 0.023414 with SlimPajama.
This is the best-behaved model in the reproduction
Across all 64 layers the hypernet cut reconstruction error 75-84% below the SparseGPT prior, with mask agreement rising to a 0.83 plateau, and no layer regressed — unlike gpt-oss-20b, where three layers ended worse than the prior and were kept anyway (the method has no keep-best guard). Greedy generation is coherent.
Scope is MLP only (mlp.gate_proj/up_proj/down_proj), matching BLADE's
two_four_experts format for this model: attention stays dense. The model is a hybrid —
48 of its 64 layers use linear_attn and 16 use self_attn — but the MLP is identical
across both layer types, so the pruned set is uniform.
Provenance
Produced from HyperPrune commit 6d093d7 with a small set of documented
patches (bias-dtype autocast, calibration loader, disk-peak reduction, and —
for 4:8 checkpoints — the N:M generalization, which the reference
implementation does not ship). See the reproduction report for the full diff.