NVIDIA-Nemotron-3-Nano-4B NVFP4 GGUF
NVFP4 (E4M3) quantization of nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16, NVIDIA's 4B-parameter dense text model with Mamba2-SSM hybrid attention and 1,048,576 token context window.
Why Is This File Bigger Than Expected?
A pure NVFP4 quant of this model should be ~2.3 GB, but this file is ~4.6 GB. Here's why:
This model uses a Mamba2-Transformer hybrid architecture — 37 out of 41 layers are Mamba2 state-space model (SSM) layers, not standard transformer attention. The CUDA kernels that run Mamba2 SSM operations (SSM_SCAN, SSM_CONV) require F32 inputs and will reject quantized weight tensors entirely.
When SSM weight tensors are stored in NVFP4, the CUDA backend refuses to process them and falls back to CPU for all 37 Mamba2 layers. Since 90% of the model is Mamba2, this means the entire model effectively runs on CPU — extremely slow.
The Workaround (Hybrid Quantization)
To fix this, the 42 SSM weight tensors (ssm_in.weight, ssm_out.weight across 37 Mamba2 blocks) are stored in F16 instead of NVFP4. Everything else — attention layers, FFN layers, embeddings, norms — stays at NVFP4.
| Layer Type | Count | Storage Format | Why |
|---|---|---|---|
| Mamba2 SSM weights | 37 layers | F16 | CUDA SSM kernels require F32/F16 input |
| Attention weights | 4 layers | NVFP4 | Standard attention CUDA ops handle quantized input |
| FFN weights | all layers | NVFP4 | Standard FFN CUDA ops handle quantized input |
| Bias/norm tensors | all layers | F32 | Always F32, too small to matter |
This adds ~2.3 GB on disk but ensures all layers run on CUDA GPU instead of falling back to CPU.
This Is a Bandaid, Not a Fix
This hybrid approach is a temporary workaround until one of the following happens:
-
llama.cpp adds NVFP4 support to CUDA SSM kernels — The Mamba2 scan and conv CUDA ops currently assert
GGML_TYPE_F32inputs. Adding NVFP4 dequantization support inside these kernels would allow pure NVFP4 storage with no performance penalty. -
NVIDIA releases native NVFP4 SSM kernels — Blackwell GPUs have hardware NVFP4 support, but the SSM-specific CUDA kernels haven't been updated to use it yet.
-
A different quantization approach is found — e.g., quantizing only the non-SSM layers and keeping SSM at higher precision by design, rather than as a post-hoc requantization.
Until then, this hybrid GGUF is the best available option for running this model with NVFP4 on Blackwell GPUs. The file is larger than ideal, but the alternative (CPU fallback for 90% of the model) is much worse.
Files
| Filename | Type | Size | Description |
|---|---|---|---|
nemotron-3-nano-4b-NVFP4.gguf | Hybrid NVFP4+F16 | ~4.6 GB | Main model weights (~9.72 BPW effective) |
Architecture
- 41 layers: 37 Mamba2 (SSM) + 4 sliding-window attention
- Hybrid pattern: M-M-M-MM-M-M*-M-M*-M-M-M*-M-M-MM*-MMM-M-M-M
- Attention blocks: 3, 12, 17, 24, 32 (head_dim=128, 8 KV heads)
- Mamba2 blocks: All others (conv kernel 4, state size 128, 96 heads)
Usage
llama-server -m nemotron-3-nano-4b-NVFP4.gguf --host 0.0.0.0 --port 8080 -ngl 99
Credits
- Original model: NVIDIA
- Quantization: FreedomAISVR