Qwen3-4B — FlatQuant W4A4KV4 with a 12K calibration prefix
September 13, 2026 prefix12k-no-kv ablation from the RECAP project.
This checkpoint quantizes Qwen/Qwen3-4B with FlatQuant using a 12,288-token dynamic prefix and a 4,096-token active calibration block.
The repository name refers to the RECAP project; KV read-geometry calibration and its auxiliary loss are disabled in this variant (kv_curvature_method=none, kv_curvature_lambda=0).
The experiment label no-kv means no KV-curvature auxiliary loss. Key and value quantization remain enabled at 4 bits.
Quantization
| Component | Precision and configuration |
|---|---|
| Linear weights | 4-bit, symmetric, per output channel; w_groupsize=-1 |
| Activations | 4-bit, asymmetric, per token; a_groupsize=-1 |
| Keys | 4-bit, asymmetric, group size 128 |
| Values | 4-bit, asymmetric, group size 128 |
| Queries | 16-bit; no low-bit query quantization |
| Scale format | FP32 |
| Quantization scope | all, across all 36 decoder layers |
Learned weight/activation clipping and learned FlatQuant transforms are enabled. This is a fake-quantization checkpoint stored as BF16 tensors, with the custom runtime applying the configured activation and KV quantization. It is not a packed INT4 checkpoint and does not imply packed INT4 memory use or integer-kernel speedups. The original 11 model files total 8,081,007,426 bytes (about 8.08 GB).
Calibration
- Data: Qwen3-4B-generated reasoning trajectories from NuminaMath-1.5; 64 training examples and 8 held-out examples.
- Dynamic prefix: 12,288 tokens; active block: 4,096 tokens; total calibration window: 16,384 tokens, with tiled position coverage.
- 32 epochs; seed 0; batch size 4; microbatch size 1; learning rate 0.005.
- Hessian weighting, CBI weighting, KV-curvature loss, SGAS joint optimization, and end-to-end KL tuning are disabled.
12K describes the calibration prefix, not the model's maximum context length. The exported configuration sets max_position_embeddings=40960. The recorded smoke test configured this context limit and checked short prefill and cached decoding; it was not a full 40,960-token quality evaluation.
See calibration_summary.json for structured settings and provenance. Calibration data and training logs are not distributed here.
Loading requirements
The architecture is Qwen3FlatQuantForCausalLM and its quantization settings live in fake_quant_config. Inference requires a compatible RECAP Qwen3 FlatQuant implementation and explicit registration of that custom architecture with vLLM. Generic Transformers loading or an unmodified vLLM installation does not reproduce this checkpoint's quantization behavior.
The local validated environment uses PyTorch 2.7.0+cu128, Transformers 4.51.3, and a vLLM 0.9.0+sm120 build. Use a vLLM build appropriate for your GPU and a compatible RECAP runtime. The evaluated runtime includes local changes beyond the recorded project Git commit; a checkout of that commit alone is not a verified loading recipe.
Keep both model-00001-of-00002.safetensors and model-00002-of-00002.safetensors, model.safetensors.index.json, flat_matrices.pth, and all configuration/tokenizer files together. The matrix file contains transforms and clipping parameters for all 36 layers and is required by the custom loader.
Validation confirmed all 398 indexed weight tensors are present, all 36 matrix records load, and the tokenizer/chat template loads locally. An existing runtime smoke test passed full checkpoint loading, short prefill, cached decode, and finite log probabilities.
License
Derived from Qwen3-4B under the Apache 2.0 license. See LICENSE. The base weights were transformed and quantized using the FlatQuant calibration recipe described above.