[!TIP] KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use
-ctk q8_0 -ctv q8_0(~half KV memory, negligible quality loss: perplexity +0.002–0.05) or-ctk q4_0 -ctv q4_0(~quarter memory, ≈7.6% perplexity increase). In Ollama:OLLAMA_KV_CACHE_TYPE=q8_0withOLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out:LLAMA_ATTN_ROT_DISABLE=1).The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.
Nemotron-3-Nano-Omni-30B-A3B-Reasoning - TurboQuant GGUF MXFP4_MOE
GGUF MXFP4_MOE quantization of Nemotron-3-Nano-Omni-30B-A3B-Reasoning (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) with TurboQuant weight method.
The MXFP4_MOE.gguf binary in this repo is loaded by llama.cpp / llama-mtmd-cli.
For multimodal inference (text + image + audio + video) pair this with the
multimodal projector: majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-mmproj-F16.
For the matched-KV stack — TurboQuant weights + TurboQuant KV-cache modifier —
For the runtime KV-cache modifier itself (weight-agnostic), see
majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant.
Quickstart
# 1. Download the GGUF + the multimodal projector
huggingface-cli download majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-GGUF-MXFP4_MOE MXFP4_MOE.gguf --local-dir ./model
huggingface-cli download majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-mmproj-F16 mmproj-F16.gguf --local-dir ./mmproj
# 2. Multimodal inference (text + image + audio + video)
llama-mtmd-cli \
-m ./model/MXFP4_MOE.gguf \
--mmproj ./mmproj/mmproj-F16.gguf \
--image cat.jpg \
-p "Describe this image in detail" \
--temp 0.6 --top-p 0.95 -n 512
# 3. Text-only inference (no mmproj needed)
llama-completion -no-cnv \
-m ./model/MXFP4_MOE.gguf \
-p "What is the capital of France?" \
--temp 0.6 --top-p 0.95 -n 256
# Disable extended reasoning (default is on):
# add `--chat-template-kwargs '{"enable_thinking": false}'`
⚠️ Do NOT use llama.cpp built against CUDA 13.2 — produces gibberish. Pin CUDA 12.x or use Metal/CPU.
Modality matrix
| Modality | Encoder | Quantization in this variant |
|---|---|---|
| Text | LLM backbone (Mamba-2 + Transformer hybrid Sparse MoE) | per the variant suffix |
| Image | CRADIO v4-H | BF16 (kept full-precision in every non-GGUF variant; GGUF uses mmproj-F16 split file) |
| Audio | Parakeet-TDT-0.6B-v2 | BF16 (same rationale) |
| Video | Parakeet-TDT-0.6B-v2 + frame sampler | BF16 (≤ 2 min, 256 frames @ 2 FPS) |
NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal MLP projectors in BF16 to preserve multimodal accuracy. We follow that convention in every quantized variant we ship.
Runtime quirks
llama.cpp
Use llama-mtmd-cli for multimodal inference; pass --mmproj mmproj-F16.gguf
(see majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-mmproj-F16).
Do NOT use CUDA 13.2 — produces gibberish. Pin CUDA 12.x or use the Metal/CPU paths.
Ollama
Text-only; multimodal is blocked because Ollama doesn't yet support the mmproj split-file pattern.
Reasoning mode
enable_thinking defaults to True. To disable extended reasoning
(e.g., for latency-sensitive cases), pass enable_thinking=False
to the chat template / generate call. No separate "no-think"
variant card exists — this is a runtime flag, not a model variant.
Quant trade-off (GGUF lane)
| Quant | Approx size | Use case | Recommendation |
|---|---|---|---|
| Q2_K | ~17 GB | Lossy, low-RAM CPU/edge | Resource-constrained inference |
| Q3_K_M | ~19 GB | Smaller-than-Q4, modest quality drop | Edge devices with ~16 GB RAM |
| IQ4_XS | ~16 GB | Importance-quant 4-bit, smaller than Q4_K_M | Best size/quality at 4-bit |
| Q4_K_M | ~23 GB | Balanced default | Recommended for most users |
| Q5_K_M | ~24 GB | Higher fidelity than Q4 | Quality-sensitive applications |
| Q6_K | ~28 GB | Approaching FP16 quality | High-fidelity CPU/edge |
| Q8_0 | ~32 GB | Near-lossless reference | Fidelity-critical work |
| MXFP4_MOE | ~17 GB | Microscaling FP4 (MoE-aware) | vLLM / transformers users |
(Current variant — MXFP4_MOE — is bolded.)
Variants in this family
(Showing 56 sibling variants under majentik/nemotron3-nano-omni-30b-*. The current variant — TurboQuant-GGUF-MXFP4_MOE — is bolded.)
| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| TurboQuant-GGUF-MXFP4_MOE | llama.cpp | ~30 GB | MXFP4 MoE quant |
About the RotorQuant / TurboQuant labels
RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured.