Qwen3-4B-NWC
Qwen/Qwen3-4B with all linear layers and the tied embedding stored in NWC (Neural Weight Compression) format: losslessly compressed BF16, decoded inside the CUDA matvec kernel. The weights are bit-exact with the original checkpoint; no quantization is involved.
| Qwen3-4B (BF16) | Qwen3-4B-NWC | |
|---|---|---|
| weights | 8.04 GB | 5.55 GB (rate 0.689) |
| VRAM in use, batch 1 | 8.10 GB | 5.67 GB |
| tokens/s, RTX 4070, CUDA graph, greedy | 45 | 55 |
| tokens/s, NVIDIA A16 (vGPU 16Q) | 16.8 | 18.2 |
| output tokens vs original (greedy, 64) | — | identical |
Usage
pip install neural-weight-compression transformers accelerate
from nwc import load_pretrained
from transformers import AutoTokenizer
model = load_pretrained("Parda21/Qwen3-4B-NWC") # no BF16 originals needed, loads straight to the GPU
tok = AutoTokenizer.from_pretrained("Parda21/Qwen3-4B-NWC")
ids = tok("Question: What is a binary tree? Answer:", return_tensors="pt").input_ids.cuda()
print(tok.decode(model.generate(ids, max_new_tokens=64, do_sample=False)[0]))
Or from the command line: python -m nwc.demo Parda21/Qwen3-4B-NWC --load --graph
Requirements: NVIDIA GPU with compute capability 8.0 or newer (Ampere, Ada, Hopper; Blackwell via PTX JIT), CUDA driver for CUDA 12.6+, PyTorch with CUDA. Batch-1 generation runs through the fused kernel; prefill (batch > 1) dequantizes into a temporary BF16 buffer and uses cuBLAS.
How it was made
from nwc import fuse, convert, save_pretrained
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", dtype=torch.bfloat16, device_map="cpu")
fuse(model); convert(model)
save_pretrained(model, "Qwen3-4B-NWC", tokenizer=tok, base_model="Qwen/Qwen3-4B")
Format and measurements: https://github.com/parda21/NWC (docs/format.md, docs/results.md). License of the weights: Apache-2.0, as the base model.