NVIDIA Nemotron 3.5 Lightning 30B-A3B - NVFP4 GGUF for llama.cpp

A GGUF repack of NVIDIA's Nemotron 3.5 Lightning 30B-A3B (NVFP4) so it runs on llama.cpp's native Blackwell (sm_120) FP4 tensor-core path. The 4-bit NVFP4 expert weights are copied bit-exact into GGUF; the remaining tensors (attention, token embedding, Mamba-2 projections, and the MTP head) are stored at Q8_0. The 4-bit weights are never re-quantized, so the result is numerically faithful to NVIDIA's checkpoint.

About the model

Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + attention model: 30B total parameters (~3B active), 52 layers of which only 6 use full attention, 128 routed experts (6 active per token) plus 1 shared expert, and a native context length of 1,048,576 tokens.

Because only 6 of the 52 layers keep a growing KV cache (the Mamba-2 and MoE layers carry a fixed-size state), the model holds its full 1M-token context on a single 24 GB GPU - something a dense model of this size cannot do.

Files

Grab the base model for normal use; add the MTP head only if you want speculative decoding.

Running it (llama.cpp)

Base model, full 1M context on a 24 GB card:

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 1048576 -fa on --cache-type-k q4_0 --cache-type-v q4_0

With the MTP head for speculative decoding (draft depth 2 is the sweet spot; it uses more memory, so reduce the context on a 24 GB card):

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -md mtp-Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 524288 -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-ngl 99 \
    --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

The MTP head is a modest, consistent speedup - roughly +7% on fast GPUs and low-double-digit percent on slower hardware (see the chart above) - at the cost of some context headroom on 24 GB. It affects throughput only, never output quality: the draft's tokens are always verified by the base model.

Quality

Evaluated on three benchmarks from the TNG benchmark suite (MMLU-Pro and terminal-bench 2.0 on distilled subsets, IFBench in full), for the released configuration (base model + external MTP head):

The repack is numerically faithful to NVIDIA's NVFP4 checkpoint by construction, and these scores are consistent with the source model within measurement noise. The benchmarks use distilled subsets with small sample counts, so single-point differences are noisy - terminal-bench in particular varies substantially run to run; treat the numbers as indicative rather than precise.

How it was made

Converted with llama.cpp's convert_hf_to_gguf.py: the routed experts, shared expert, and LM head stay NVFP4 (repacked bit-exact); the token embedding, attention projections, Mamba-2 in/out projections, and the MTP head are stored at Q8_0 (near-lossless). No dequantize-and-requantize step, so the vendor's calibrated 4-bit weights are preserved exactly.

License and attribution

Released under the OpenMDW License Agreement, version 1.1, inherited from the base model. Model materials Copyright (c) 2026 NVIDIA Corporation & Affiliates.

This is a format repackage (not a retrain) of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4. The only changes are quantization / format conversion to GGUF, documented under How it was made.

Attribution: please credit NVIDIA for the Nemotron 3.5 Lightning model and its NVFP4 (TensorRT Model Optimizer) export, alongside this repackage.

Trademarks: "NVIDIA" and "Nemotron" are trademarks of NVIDIA Corporation. The OpenMDW license does not grant trademark rights; these names are used here only to identify the upstream model. This repackage is not affiliated with, sponsored by, or endorsed by NVIDIA.