NVIDIA-Nemotron-3.5-Lightning-30B-A3B — GGUF

GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, built with llama.cpp ba360ef. Every claim below is measured; raw logs are in the metrics repo.

AD- marks an Atomic Dynamic layout: the two expert tensors are assigned explicitly instead of left to a preset, and the name carries the assignment — AD-<down_exps>-<up_exps>, collapsed to one name when both match. Everything outside the experts is Q8_0; it is 7% of the model, so spending bits there is cheaper than reasoning about it.

Files

Measured on eval_neutral held-out, KL divergence against the Q8_0 reference. tok/s is generation on a single RTX 5090, full offload.

¹ what llama.cpp actually writes after its fallback rules, not what the preset asked for.

Pick AD-IQ4_NL unless you have a reason not to: it matches Q5_K_M on every metric while being 6.9 GB smaller, and it is the smallest quant on this model that still benefits from calibration.

Where the AD layouts win

Same quality, 6.9 GB less. And against ggml-org's NVFP4 build at effectively identical size:

4.6× lower divergence at the same size. quantize_nvfp4 discards the importance matrix (GGML_UNUSED(quant_weights) in ggml-quants.c), so an NVFP4 file cannot be calibrated at all. That is the whole gap. NVFP4 does win on prompt processing — 14460 vs 11809 t/s at pp512, from native Blackwell FP4 kernels — so if you are prefill-bound rather than quality-bound it remains the better pick.

What we could not deliver: a usable 16 GB quant

Q2_0 (block 64, 2.25 bpw) is the only sub-4.5 bpw type that survives this model's shapes, and it is calibration-blind — quantize_q2_0 never forwards the imatrix. Measured, it does not hold up:

A top-1 agreement of 33% means the model picks a different token two times out of three. These are published under experimental/ for reproducibility and should not be used for real work. There is currently no good 16 GB option for this model, from anyone — the smallest file here that we would stand behind is AD-IQ4_NL at 19.7 GB. Fixing that needs a block-64 superblock variant of the K-quant family in llama.cpp, which does not exist yet.

Why there is no 2-bit or 3-bit k/i quant

K- and I-quants store weights in superblocks of 256: 256 weights split into 16 sub-blocks of 16, each carrying a 6-bit scale and 6-bit minimum, all normalised against two fp16 super-scales. That hierarchy is what makes 2- and 3-bit quantization viable — at 2 bits you cannot afford an fp16 scale every 32 weights, since the scale alone would cost 0.5 bpw.

The price is that a tensor row must divide by 256. This model's expert tensors do not:

ffn_down_exps   ne0 = 1856 = 64 × 29      1856 % 256 = 64
ffn_up_exps     ne0 = 2688 = 128 × 21     2688 % 256 = 128

Together they are 93% of the model. When a row does not divide, llama.cpp's tensor_type_fallback() substitutes a block-32 type and logs a warning — it does not fail, and the file keeps the requested name.

This is why published IQ2_XXS, IQ3_XXS, Q2_K and Q3_K_M files for this model all land around 18–19 GB, and why Q6_K and Q8_0 come out the same size. Every published IQ2_XXS for this model is a 4.5 bpw file.

Types that survive these shapes, and whether they read the imatrix:

Not a defect in the model — NVIDIA's own NVFP4 uses group 16/64, which these shapes divide cleanly. It is specific to llama.cpp's k/i family.

To check any model before you quantize it:

python3 -c "
import json,sys; c=json.load(open(sys.argv[1]))
for k in ('hidden_size','intermediate_size','moe_intermediate_size'):
    v=c.get(k)
    if v: print('%-22s %6d  %%256=%3d' % (k,v,v%256))
" config.json

Speculative decoding: all three paths, measured

The model supports three. MTP is inside every file here; DFlash and DSpark are sidecars.

# MTP — no extra file needed
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  --spec-type draft-mtp -ngl 99 -c 8192

# DFlash — sidecar, auto-discovered by the dflash- prefix
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  -md dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf \
  --spec-type draft-dflash -ngl 99 -ngld 99 -c 8192

Generation tok/s, single RTX 5090, full offload, identical prompt and seed, 400 tokens:

On a fast GPU with full offload, speculation costs throughput on every quant — MTP lands at 65–78% of baseline, DFlash at 47–55%. Draft quality is not the problem: MTP accepts 61–78% of drafted tokens with a mean accepted length above 3.

The cost is the rollback. This is a hybrid Mamba2 model, so discarding a rejected draft means restoring SSM state, and the log shows 47.6 MiB copied per speculation step. We expected larger quants to flip the result, since the target gets more expensive per token while the checkpoint cost stays fixed. They did not — the ratio is flat across a 3× size range, so the copy scales with the work too.

Untested here, and where speculation should still pay: partial offload and CPU inference, where a token costs far more than 47.6 MiB of VRAM-to-VRAM copy. Measure on your hardware before enabling it.

DSpark does not load. llama.cpp requires a conf_proj.weight of shape {n_embd + dspark_markov_rank, 1} (src/models/dflash.cpp:94), but NVIDIA's DSpark checkpoint ships markov_head.markov_w1 / markov_w2 and no confidence projection, so it fails with check_tensor_dims: tensor 'conf_proj.weight' not found. The converted file is published anyway in case someone fixes the runtime side.

Converting either sidecar needs --target-model-dir, since neither carries a tokenizer:

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash \
  --outtype bf16 --target-model-dir NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outfile dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf

imatrix

Calibrated on AtomicChat/calib-corpora, recipe nemotron-3.5-lightning: 4.72M tokens over 3,035 documents — agentic 26.5%, code 19.1%, reasoning 15.9%, long-context 12.7%, multilingual 12.8%, vocabulary sweep 5.8%, structured 4.0%, graphics 3.2%.

Rendered through this model's own chat template, so <|im_start|>, <think>, <tool_call> and <tool_response> appear as the single special tokens the model actually sees. Run llama-imatrix with --parse-special, or that markup is tokenized as literal punctuation and ~40% of the corpus calibrates nothing.

Past 8.5k chunks no tensor changes measurably, so more calibration data of this kind cannot change the quantizer's decisions. Reproduce with tools/converge.py in the dataset repo.

All 128 experts in all 46 expert tensors receive routing — 0 dead experts at every corpus size tested, including 823 chunks. Expert coverage is not what a larger corpus buys.

The MTP block (blk.52) is never executed in a normal forward pass, so it collects no imatrix data at any corpus size. It is pinned to q4_0 in every file: --tensor-type 'blk\.52\.=q4_0'.

Architecture

52 layers: 23 Mamba2, 23 MoE, 6 attention (indices 5, 12, 19, 26, 33, 42). 128 experts, 6 active plus 1 shared. hidden_size 2688, moe_intermediate_size 1856, squared-ReLU FFN — experts have up and down only, no gate. GQA 32/2, head_dim 128, rope_theta 10000, no sliding window, context 262144.

With only 6 attention layers at 2 KV heads the KV cache costs about 6 KB per token, and Mamba2 state is a constant ~50 MB per sequence regardless of context length. Long context is unusually cheap here — budget for weights, not for cache.

Running

llama-server -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf -ngl 99 -c 32768

Needs llama.cpp b10326 or newer for nemotron_h_moe. Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Reproducing

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outtype bf16 --outfile Nemotron-BF16.gguf

llama-imatrix -m Nemotron-Q4_0.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --save-frequency 500

llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.52\.=q4_0' \
  --tensor-type 'ffn_down_exps=iq4_nl' \
  --tensor-type 'ffn_up_exps=iq4_nl' \
  Nemotron-BF16.gguf Nemotron-AD-IQ4_NL.gguf Q8_0 64

-b 8192 is not cosmetic: at the default batch the imatrix run takes 2h12m instead of 1h22m on the same hardware.