Nemotron-3-30B-Nano-Omni MXFP4 MoE GGUF
MXFP4 MoE quantization of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, NVIDIA's 30B MoE (~3B active) reasoning model with Mamba2-SSM hybrid architecture.
Why Is This File Bigger Than Expected?
This model uses a Mamba2-Transformer hybrid MoE architecture — 23 out of 52 layers are Mamba2 state-space model (SSM) layers. The CUDA kernels that run Mamba2 SSM operations (SSM_SCAN, SSM_CONV) require F32 inputs and will reject quantized weight tensors entirely.
When SSM weight tensors are quantized, the CUDA backend falls back to CPU for those layers. Since 44% of the model is Mamba2, this significantly impacts performance.
The Workaround (Hybrid Quantization)
The 46 SSM weight tensors (ssm_in.weight, ssm_out.weight across 23 Mamba2 blocks) are stored in F16 instead of the quantized format. Everything else stays at the original quantization.
This Is a Bandaid, Not a Fix
This hybrid approach is a temporary workaround until llama.cpp adds quantized support to CUDA SSM kernels (SSM_SCAN, SSM_CONV). The kernels are currently hardcoded to F32 (ssm_conv_f32, ssm_scan_f32 in the CUDA backend).
Files
| Filename | Type | Size | Description |
|---|---|---|---|
nemotron-3-30b-MXFP4_MOE.gguf | Hybrid MXFP4+F16 | ~17.9 GB | Main model weights (~4.76 BPW effective) |
mmproj-nemotron-3-30b-f16.gguf | F16 | ~1.5 GB | SigLIP vision encoder |
Quantization Details
MXFP4 MoE (Hybrid)
- MoE expert weights: MXFP4 (routed + shared experts)
- Dense non-expert weights: Q8_0 (attention, norms, router)
- SSM weight tensors: F16 (CUDA SSM kernel compatibility)
- Bias/norm tensors: F32
This is the recommended format for this model — MoE-aware routing processes only 6 experts per token instead of all 128.
Architecture
- 52 layers: 23 Mamba2 (SSM) + 23 MoE FFN (128 experts, top-6) + 6 attention
- MoE: 128 routed experts + 1 shared expert, top-6 routing
- Vision: SigLIP ViT (30 blocks, 1280 hidden)
- Context: 262K tokens
Usage
llama-server -m nemotron-3-30b-MXFP4_MOE.gguf --mmproj mmproj-nemotron-3-30b-f16.gguf --host 0.0.0.0 --port 8080 -ngl 99
Credits
- Original model: NVIDIA
- Quantization: FreedomAISVR