DeepSeek-R1-Distill-Llama-70B-NVFP4-GGUF
NVFP4 GGUF of DeepSeek-R1-Distill-Llama-70B, optimized for NVIDIA Blackwell GPUs (RTX 5090, RTX PRO 6000, B200).
Overview
| Property | Value |
|---|---|
| Base model | deepseek-ai/DeepSeek-R1-Distill-Llama-70B |
| Quantization | NVFP4 (W4A4 — FP4 E2M1 weights, FP8 E4M3 block scales, FP32 global scales) |
| Format | GGUF (GGML_TYPE_NVFP4, type 40) |
| Size | 42.7 GB |
| BPW | ~4.84 bits per weight |
| Requires | NVIDIA Blackwell GPU (SM 120) for native FP4 tensor core acceleration |
| Speed | 33.6 tok/s generation on RTX PRO 6000 96GB (single GPU) |
Quantization Pipeline
This model was quantized using a custom pipeline:
- Quantization tool: vllm-project/llm-compressor v0.10.0.2
- Calibration: 128 samples from
ultrachat-200k(train_sft split), max_seq_length 2048 - Recipe:
QuantizationModifier: targets: [Linear] ignore: [lm_head] scheme: NVFP4 - GGUF conversion: llama.cpp PR #21095 branch with two critical fixes:
- Tensor renaming:
weight_packed→weight,weight_global_scale→weight_scale_2,input_global_scale→input_scale - Scale inversion: llm-compressor stores
weight_global_scaleas a large divisor; ggml expects a small multiplier. Fix: store1.0 / weight_global_scalein the GGUF.scaletensor.
- Tensor renaming:
Important: Scale Inversion Fix
If you're converting your own llm-compressor NVFP4 checkpoints to GGUF, you must invert the per-tensor global scale. llm-compressor stores scales like 4320.0 (divide by this to dequantize), while ggml's convention expects 0.000231 (multiply by this). Without this fix, the model loads and runs but produces garbage output.
In convert_hf_to_gguf.py, change:
self._write_scale_tensor(new_name.replace(".weight", ".scale"), scale2)
to:
self._write_scale_tensor(new_name.replace(".weight", ".scale"), 1.0 / scale2.float())
Provenance
| Component | Source |
|---|---|
| Original base model | deepseek-ai/DeepSeek-R1-Distill-Llama-70B |
| NVFP4 checkpoint | PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4 |
| GGUF conversion tooling | llama.cpp (PR #21095) |
| Quantization hardware | NVIDIA RTX PRO 6000 Blackwell 96GB |
Usage
llama-server
./llama-server \
-m DeepSeek-R1-Distill-Llama-70B-NVFP4.gguf \
--host 0.0.0.0 \
--port 8082 \
-ngl 999
llama-cli
./llama-cli \
-m DeepSeek-R1-Distill-Llama-70B-NVFP4.gguf \
-ngl 999 \
-c 8192 \
--jinja \
-p "What is the meaning of life?"
Benchmarks
Tested on RTX PRO 6000 Blackwell 96GB, llama.cpp b9037:
| Metric | Value |
|---|---|
| Prompt eval | 196.5 tok/s |
| Generation | 33.6 tok/s |
| Context | 131072 (default) |
| Parallel slots | 4 (default) |
| VRAM usage | ~80 GiB (model + KV cache at 131K context) |
Comparison
| Backend | tok/s |
|---|---|
| llama.cpp (this GGUF) | 33.6 |
| vLLM 0.19.0 (compressed-tensors) | 25.0 |
Known Issues
- This is a community conversion, not an official upstream release.
- The NVFP4 GGUF ecosystem is still maturing. See llama.cpp Discussion #22042 for the ongoing discussion about NVFP4 scale handling.
- Blackwell-native tensor core dispatch (PR #22196) may improve performance further when merged.
Credits
Quantized and converted by PiehSoft (William Pieh). Built on work by the llama.cpp community, particularly PR #21095 for NVFP4 GGUF conversion support.