Qwen3.8-27B-DSpark-NVFP4
Weight-only NVFP4 DSpark speculative draft head for nvidia/Qwen3.8-27B-NVFP4, validated with lithos-metal on Apple silicon. This head runs alongside the target model.
Contents and format
- 37 matrices stored in NVFP4.
- 1,136,258,502 bytes of safetensors weights (1.136 GB, decimal).
- FP4 E2M1 weights, two codes per
U8byte, low nibble first. - One
F8_E4M3block scale per 16 input columns and oneF32tensor scale per matrix. - Tensor names:
*.weight,*.weight_scale, and*.weight_scale_2(ModelOpt-style storage). - Dequantization:
E2M1(code) * E4M3(block_scale) * tensor_scale. - Markov W1 stays BF16; normalization and confidence parameters retain their original precision.
- Weight-only arithmetic with the runtime's BF16 activations and FP32 accumulation/state; no activation calibration or NVFP4 activation quantization is declared.
- Sharded, standard safetensors plus an index; no hardware-specific
weights.packis required in the download.
This head shares token embeddings and the vocabulary projection with the selected target model. Five draft layers, hidden size 5120, intermediate size 17408, 32 query heads, eight KV heads, head dimension 128, and Markov rank 256 are preserved. The checkpoint serves seven proposals.
Run with lithos-metal
Loading is validated with lithos-metal, on a 40-core Apple M5 Max with 48 GB memory. The target provides the tokenizer and chat template. The NVFP4 tensor layout alone does not establish compatibility with stock Transformers, SGLang, vLLM, or another DSpark loader. This checkpoint uses the lithos-metal DSpark loader and does not require remote Python code.
Use lithos-metal v0.1.2
for a reproducible runtime. Follow its
installation and build instructions.
This release includes the NVFP4 embedding tensor-scale fix, draft attention specialization
compatibility fix, and the validated NVFP4 serving recipe for the matching 35B draft shapes.
The included Apache-2.0 lithos-metal-runtime.patch is retained
for older runtimes and is not needed with this release.
Once the engine and its serving dependencies are installed:
python -m monolith.serve \
--model nvidia/Qwen3.8-27B-NVFP4 \
--draft LithosAI/Qwen3.8-27B-DSpark-NVFP4 \
--draft-quantization nvfp4 \
--draft-block-size 7 \
--max-context 32768 \
--host 127.0.0.1 --port 8000
Model IDs or local checkpoint directories are accepted. The runtime builds and reuses a local chip-layout pack automatically. Already quantized matrices are read directly, without another quantization pass.
Validation and performance
Both standard safetensors reading and a complete pack round trip are checked. Repacking this
export reproduces the previously quality-tested draft pack byte for byte, including all
weight codes, block/tensor scales and auxiliary tensors. See validation.json and conversion.json
for the pack SHA-256, tensor inventory and quantization policy.
A fresh pack build also succeeds with both NVFP4 CPU quantize and dequantize functions disabled, and matches the same pack hash. Loading these Hub weights therefore needs only layout packing, not dequantization or requantization. Two real Chat Completions requests per export return HTTP 200, eight verification rows, and identical repeated 48-token outputs.
On 40-core M5 Max, the recorded paired 128-context comparison measured a draft-stage median of 9.552 ms BF16 versus 4.392 ms NVFP4 and a complete-round median of 48.427 versus 43.188 ms. Nine alternating samples followed five warmups. Both versions accepted all seven proposals on that timing fixture; a round includes target verification, acceptance/commit, and the next draft block. Loading, compilation, prefill and HTTP work are excluded.
The subsequent NVFP4 kernel refinement measured a 4.186 ms draft span and 42.611 ms complete round in a separate run. These later measurements are not a paired comparison with the BF16 values above.
All 512 generated target tokens across four 128-token prompts matched the BF16 reference. Acceptance histories differed; this small prompt set does not establish a general quality or acceptance advantage. The refined NVFP4 run used 150 rounds versus 152 for the BF16 baseline. Historical study: draft refinement.
License
See WEIGHT_NOTICES.md for the applicable weight license terms.
The exporter and runtime compatibility patch are derived from the Apache-2.0 lithos-metal project; see LICENSE.runtime.