YuE2-3B · OrbitQuant W4A4
Music generation from style and lyrics (m-a-p/YuE2-3B) with packed 4-bit transformer weights and activations. Output: 48 kHz stereo.
Model
- 392 transformer projections in OrbitQuant W4A4 (224 packed modules). Embeddings, norms, auxiliary projections and the audio VAE stay in source precision; output heads are INT8 or W4A4 only in the profiles below.
- Checkpoint 3.04 GB vs 7.26 GB BF16 (−58.2%). Separate 0.53 GB VAE unchanged. No fine-tuning or distillation.
Run
hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --mode fast --output out
One request per process is the default. A long-lived process serving many requests adds --warm-compile (compiled VAE decoder: 0.2 s instead of 0.7 s per request after a one-time 2 s cache load).
--mode | Changes | Logits |
|---|---|---|
off | reference decode path | reference |
exact | lookahead decode loop, ABC head cropped to the sampler's classes, model resident during VAE decode, compiled VAE decoder | bit-identical to off |
fast | + KV handoff to NAR, compiled NAR pointwise ops, INT8 output heads (DP4A GEMV) | KL 0.0015 semantic / 0.0002 ABC |
lowmem | + INT8 token embeddings | as fast |
turbo | lowmem with W4A4 output heads | KL 0.016 / 0.0009 |
Requirements: Linux x86_64, Python 3.12, PyTorch 2.10.0 + CUDA 12.8; the OrbitQuant 0.9.8 wheel is bundled in src/wheels, native kernels 1.0.5 under src/runtime/kernels. Kernels: SM89 (validated on RTX 4090) and SM120 (built, not run). The first request compiles the VAE decoder (~25 s, cached afterwards).
Measurements
RTX 4090 (48-core host), identical input and seed. "Warm" is the second request of a long-lived process with --warm-compile; "process" is one src/run.py call from interpreter start to the written files (imports and model load included). Peaks are for the whole process.
| Mode | Audio | Warm request | audio s / wall s | Whole process | NVML peak | torch peak |
|---|---|---|---|---|---|---|
off | 168.9 s | 17.94 s | 9.42 | — | 5.39 GiB | 4.31 GiB |
exact | 168.9 s | 15.85 s | 10.66 | 21.6 s | 5.33 GiB | 4.05 GiB |
fast | 140.7 s | 12.26 s | 11.47 | 18.0 s | 4.64 GiB | 3.37 GiB |
lowmem | 152.2 s | 13.49 s | 11.28 | 19.3 s | 4.47 GiB | 3.37 GiB |
turbo | 178.2 s | 15.57 s | 11.44 | 21.7 s | 4.71 GiB | 3.56 GiB |
exact keeps the off tokens and logits bit for bit; its waveform differs from off only at FP16 rounding in the VAE. INT8/W4A4 heads change sampling at the same seed, so compare those modes by audio seconds per wall second. Of a whole process, about 4 s are Python and library imports plus model load; the rest is generation. Wall time also depends on the host CPU: on a slow 4-core host the same exact request took 17.7 s and off 28.3 s. Raw data: evaluation/profiles/.
Files
src/: runtime, kernels and kernel sources; src/kernel-binaries.json lists the binaries per architecture, src/runtime-lock.json pins versions, SHA256SUMS covers every file. examples/: prompt file and WAV demos (BF16, W4A4, fast INT8 KV).
License: CC BY-NC 4.0 for the weights (inherited from M-A-P / YuE2); code components keep their own licenses. No commercial rights.