Qwen3-0.6B-MYPTQ-w4g128
4-bit weight-only quantization of Qwen/Qwen3-0.6B: group size 128 along in_features with integer zero-points, plus an INT4-packed ONNX graph (com.microsoft.MatMulNBits). The 28 decoder layers shrink 3.85x (840 -> 218 MB, 4.16 bit/weight); the tied embedding is left at 16-bit.
Built with MYPTQ, which holds the pipeline, the ablations and the full report.
Usage
model.safetensors stores the codes and zero-points packed two per byte, so plain
AutoModelForCausalLM.from_pretrained cannot read it; load_realquant_model rebuilds the
RealQuantLinear modules and unpacks them.
import torch
from huggingface_hub import snapshot_download
from quantlinear import load_realquant_model # https://github.com/junuke/MYPTQ
local = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128")
model, tokenizer = load_realquant_model(local, dtype=torch.float16, device="cuda")
ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()
print(tokenizer.decode(model(input_ids=ids).logits[0, -1].argmax())) # ' Paris'
The ONNX graph keeps its weights INT4-packed and needs ONNX Runtime >= 1.16. Its tensors live in
onnx/model.onnx.data, which must sit next to onnx/model.onnx. Download the folder with
local_dir=: the default cache stores files as symlinks into its blob store, and onnxruntime rejects
external data whose real path leaves the model directory.
import numpy as np, onnxruntime as ort
graph = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128", allow_patterns=["onnx/*"], local_dir="onnx_model")
sess = ort.InferenceSession(f"{graph}/onnx/model.onnx",
providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
logits = sess.run(["logits"], {"input_ids": ids.cpu().numpy().astype(np.int64)})[0]
Activations are fp16. The graph takes input_ids [batch, seq], returns logits [batch, seq, vocab],
and builds its causal mask internally (no KV cache input).
Measured results
1. Accuracy and general quality (fp16; MMLU: lm_eval mmlu, limit=0.1, max_length=512, batch_size=16; C4: 60 x 512 tokens, out of domain)
| Model | Body weights | MMLU acc | vs baseline | C4 ppl |
|---|---|---|---|---|
| Qwen3-0.6B baseline | fp16 | 0.4000 | - | 32.51 |
| naive W4 (RTN g128, min/max, data-free) | 4-bit | 0.3483 | -5.17%p (-12.9%) | 39.51 (+21.5%) |
| SmoothQuant init + W4 (untrained) | 4-bit | 0.3524 | -4.76%p (-11.9%) | 36.45 (+12.1%) |
| MYPTQ (smooth + compensation + clip, CE+KD trained) | 4-bit | 0.4294 | +2.94%p (+7.3%) | 32.89 (+1.2%) |
2. ONNX conversion quality (com.microsoft.MatMulNBits, INT4 packed; reference = PyTorch fp32 QuantLinear model)
| ONNX graph | Size | SNR (16 x 512 tokens, wikitext2 test) |
|---|---|---|
| fp32 graph (exactness check) | 820.1 MB | 117.9 dB (min 116.8), top-1 100.00% |
| fp16 graph (deliverable) | 516.5 MB | 52.6 dB (min 51.8), top-1 99.61% |
3. Model size
| Qwen3-0.6B (bf16) | MYPTQ W4 g128 | ratio | |
|---|---|---|---|
| Body weights (28 layers; the assignment's compression metric) | 840.1 MB | 218.3 MB (4-bit codes + fp16 scale + 4-bit zero, 4.16 bit/weight) | 3.85x |
| Embedding (+ tied lm_head, excluded, kept 16-bit) | 296.8 MB | 296.8 MB | 1.00x |
| Total | 1136.9 MB | 515.1 MB | 2.21x (45.3% of original) |
| Files on disk | - | HF safetensors 515.2 MB / ONNX fp16 516.5 MB |
Summary
| Metric | Value |
|---|---|
| MMLU (baseline -> MYPTQ) | 0.4000 -> 0.4294 (+2.94%p (+7.3%)) |
| C4 perplexity (baseline -> MYPTQ) | 32.51 -> 32.89 (+1.2%) |
| Body compression | 3.85x (840 -> 218 MB) |
| Total model size | 45.3% of original |
| ONNX SNR fp32 / fp16 graph | 117.9 dB / 52.6 dB |