Qwen3.6-35B-A3B-DSpark-NVFP4
Weight-only NVFP4 DSpark speculative draft head for nvidia/Qwen3.6-35B-A3B-NVFP4, validated with lithos-metal on Apple silicon. This head runs alongside the target model.
Contents and format
- 46 matrices stored in NVFP4.
- 952,165,546 bytes of safetensors weights (0.952 GB, decimal).
- FP4 E2M1 weights, two codes per
U8byte, low nibble first. - One
F8_E4M3block scale per 16 input columns and oneF32tensor scale per matrix. - Tensor names:
*.weight,*.weight_scale, and*.weight_scale_2(ModelOpt-style storage). - Dequantization:
E2M1(code) * E4M3(block_scale) * tensor_scale. - Markov W1 stays BF16; normalization and confidence parameters retain their original precision.
- Weight-only arithmetic with the runtime's BF16 activations and FP32 accumulation/state; no activation calibration or NVFP4 activation quantization is declared.
- Sharded, standard safetensors plus an index; no hardware-specific
weights.packis required in the download.
The checkpoint-owned token embedding and frozen vocabulary projection are included and quantized. Six draft layers, hidden size 2048, intermediate size 6144, 32 query heads, eight KV heads, head dimension 128, and Markov rank 256 are preserved. The checkpoint supports eight proposals; lithos-metal defaults to seven proposals plus the anchor, or eight verification rows, without changing the stored checkpoint configuration.
Run with lithos-metal
Loading is validated with lithos-metal, on a 40-core Apple M5 Max with 48 GB memory. The target provides the tokenizer and chat template. The NVFP4 tensor layout alone does not establish compatibility with stock Transformers, SGLang, vLLM, or another DSpark loader. This checkpoint uses the lithos-metal DSpark loader and does not require remote Python code.
Use lithos-metal v0.1.2
for a reproducible runtime. Follow its
installation and build instructions.
This release includes the NVFP4 embedding tensor-scale fix, draft attention specialization
compatibility fix, and the validated NVFP4 serving recipe for the matching 35B draft shapes.
The included Apache-2.0 lithos-metal-runtime.patch is retained
for older runtimes and is not needed with this release.
Once the engine and its serving dependencies are installed:
python -m monolith.serve \
--model nvidia/Qwen3.6-35B-A3B-NVFP4 \
--draft LithosAI/Qwen3.6-35B-A3B-DSpark-NVFP4 \
--draft-quantization nvfp4 \
--draft-block-size 7 \
--max-context 32768 \
--host 127.0.0.1 --port 8000
Model IDs or local checkpoint directories are accepted. The runtime builds and reuses a local chip-layout pack automatically. Already quantized matrices are read directly, without another quantization pass.
Validation and performance
Both standard safetensors reading and a complete pack round trip are checked. Repacking this
export reproduces the previously quality-tested draft pack byte for byte, including all
weight codes, block/tensor scales and auxiliary tensors. See validation.json and conversion.json
for the pack SHA-256, tensor inventory and quantization policy.
A fresh pack build also succeeds with both NVFP4 CPU quantize and dequantize functions disabled, and matches the same pack hash. Loading these Hub weights therefore needs only layout packing, not dequantization or requantization. Two real Chat Completions requests per export return HTTP 200, eight verification rows, and identical repeated 48-token outputs.
On 40-core M5 Max, 11 alternating pairs after three warmups measured:
| Context | BF16 draft | NVFP4 draft | BF16 full round | NVFP4 full round |
|---|---|---|---|---|
| 128 | 5.62 ms | 3.00 ms | 27.42 ms | 25.17 ms |
| 4K | 6.56 ms | 3.89 ms | 30.07 ms | 27.49 ms |
| 8K | 7.25 ms | 4.71 ms | 32.29 ms | 29.50 ms |
| 16K | 9.32 ms | 6.76 ms | 36.75 ms | 34.43 ms |
| 32K | 13.91 ms | 11.27 ms | 48.76 ms | 46.38 ms |
A full nonterminal round verifies eight rows, commits state, and generates the next seven proposals. Each precision receives its own real prefill; both accept all seven proposals on this repeated hash-map timing fixture. This acceptance is not an estimate for arbitrary requests. Loading, compilation, prefill, HTTP and correctness readback are excluded.
Quality evaluation used 16 prompts × 96 generated tokens, covering code, math, explanations, writing, Chinese and Spanish. All 16 generated sequences matched the BF16 draft pipeline exactly. At 64 matched prefixes, mean accepted proposals were 3.921875/7 BF16 versus 3.859375/7 NVFP4, retaining 98.4% of the baseline's mean acceptance. Seven proposals plus the anchor and greedy decoding were used throughout. These finite comparisons qualify the conversion relative to the existing BF16 engine, not general model accuracy.
Known independent target/HF and plain/speculative numerical differences in the experimental 35B pipeline remain open; see the model support notes. The quantized embedding gather must apply its tensor scale; omitting it causes severe acceptance loss and is not a valid evaluation of this export.
License
The weights use Apache-2.0; see LICENSE and WEIGHT_NOTICES.md.
The exporter and runtime compatibility patch are derived from the Apache-2.0 lithos-metal project; see LICENSE.runtime.