Qwen2.5-Coder-32B-Instruct-BitNet-1.58b

Architecture: 32 Billion Parameters | BitNet 1.58-bit Ternary Quantization

IMPORTANT: Parameter Count Display

HuggingFace displays "9B params" because it counts packed bytes, not actual parameters. This model has the full 32B parameter Qwen2.5-Coder architecture. The weights are stored as ternary values ({-1, 0, +1}) packed 4 per byte, which reduces storage to 9.6 GB but preserves all 32 billion parameters.

Overview

This is an experimental BitNet 1.58-bit quantization of the Qwen2.5-Coder-32B-Instruct model using absmean scaling with group-wise quantization. The model stores weights as ternary values ({-1, 0, +1}) packed 4 values per byte.

This is research/experimental work. Quality and performance have not been formally benchmarked.

Specifications

Formats

GGUF Compatibility Note: The GGUF conversion is experimental. Our BitNet quantization uses group size 64, while TQ2_0 uses 256-element blocks. This may cause compatibility issues with some inference engines. The SafeTensors format is the primary supported format.

Quantization Method

Algorithm

1. Reshape weights into groups of 64
2. Compute per-group scale: scale = mean(|weights|)
3. Normalize and round to nearest ternary: q = round(w / scale) clamped to {-1, 0, +1}
4. Map to unsigned: {-1, 0, +1} → {0, 1, 2}
5. Pack 4 values per byte: v0 + v1*3 + v2*9 + v3*27

Tooling

• Quantization: Custom Rust tool using Candle
• GGUF Conversion: llama.cpp convert_hf_to_gguf.py

Hardware Used

• GPU: NVIDIA RTX 5080 (16GB VRAM)
• Quantization time: ~369 seconds (streaming mode)
• Memory: Streaming mode with CPU fallback for large tensors (>3GB threshold)

Usage

With Ollama/llama.cpp (experimental)

# llama.cpp (GGUF format - experimental, may have issues)
./llama-cli -m qwen2.5-coder-32b-TQ2_0.gguf -p "Write a Python function:"

Unpacking Weights (Python)

def unpack_ternary(packed_byte):
    """Unpack 4 ternary values from byte."""
    values = []
    val = packed_byte
    for _ in range(4):
        values.append((val % 3) - 1)  # {0,1,2} → {-1,0,+1}
        val //= 3
    return values

Limitations

• Quality not benchmarked - May have significant degradation vs original
• Requires custom runtime - Standard transformers doesn't support ternary weights
• Experimental - Not intended for production use without evaluation
• GGUF keeps embeddings/lm_head at F16, hence larger than SafeTensors
• HuggingFace may show incorrect param count due to packed storage

License

Apache 2.0 (inherited from Qwen2.5-Coder-32B-Instruct)

Citation

@misc{qwen-coder-32b-bitnet-2025,
  title={Qwen2.5-Coder-32B-BitNet-1.58b: Experimental BitNet Quantization},
  author={Tzervas},
  year={2025},
  url={https://huggingface.co/tzervas/qwen2.5-coder-32b-bitnet-1.58b}
}