NVIDIA Nemotron 3.5 Lightning 30B-A3B — APEX GGUF
Imatrix-guided, measured-allocation APEX quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B — a Nemotron-H hybrid: 52 layers of which only 6 are full attention, the rest split between Mamba2 SSM and MoE (128 routed experts + shared expert, ~3B active of 31.6B).
Three things distinguish these from the other GGUFs of this model:
- The MTP head is included. The checkpoint ships a multi-token-prediction head; most quants drop it, which makes speculative decoding impossible. These keep it (53 blocks, not 52).
- Built with an importance matrix, generated on this architecture specifically. The others are stock conversions.
- Per-tensor bit allocation is measured, not assumed — each tensor was probed for how much output KL it actually costs at each candidate width, and the budget spent accordingly.
Files
| tier | size | bpw | wikitext PPL | vs bf16 | for | status |
|---|---|---|---|---|---|---|
| mini | 13.66 GiB | 3.56 | 7.843 | +11.7% | 16 GB card with the full 128k context | available |
| compact | 16.91 GiB | 4.42 | 7.252 | +3.3% | the value pick — 5 GB smaller for 2.8% | uploading |
| i-quality | 21.53 GiB | 5.62 | 7.053 | +0.46% | best quality; 24 GB card or unified memory | uploading |
| bf16 reference | 61.32 GiB | 16.0 | 7.021 | — | (not hosted — measured as the baseline) | — |
mini is up now; compact and i-quality are still being uploaded. All three are built, measured and gated — the numbers above are from the finished files — but only what the file listing shows is downloadable yet.
nemotron-lightning.imatrix (56 MB) is included so you can build your own tiers.
Reproducing the perplexity numbers
llama-perplexity -m <tier>.gguf -f wiki.test.raw
wiki.test.raw is the unmodified WikiText-2 raw test split — 1,292,013 bytes, 241,211
words, the file llama.cpp's own perplexity documentation uses, so these numbers are directly
comparable to anyone else's. From
wikitext-2-raw-v1, which shares its
test split with WikiText-103. Identical settings across all four rows above; default context.
The calibration corpus is not WikiText. The imatrix was built on general prose and scientific text, deliberately, so the reported perplexity is measured on data the quantization never saw. Calibrating on WikiText train and then scoring on WikiText test flatters the result — the splits come from the same distribution — and it is an easy mistake to make, since the obvious calibration file to reach for is often exactly that.
Why mini fits a 16 GB card when a 30B usually doesn't
This model spends 6 KiB per token of KV cache — only 6 of 52 layers are attention, and the Mamba state is constant-size regardless of context length. So the full 128k context costs 0.75 GiB:
13.66 GiB weights + 0.75 GiB KV @ 128k = 14.41 GiB
For comparison, a conventional 35B-class MoE at ~82 KiB/token would need 10.5 GiB for that same context — the cache alone would not fit the card, let alone the weights. That pairing is the reason this model is interesting at this size point.
Quality gate
Every tier published here passed all three checks. Nothing is uploaded that did not.
| tier | PPL ratio (bar: ≤1.50) | coherent generation | chained tool-calling |
|---|---|---|---|
| i-quality | 1.00 | pass | 5/5 |
| compact | 1.03 | pass | 5/5 |
| mini | 1.12 | pass | 5/5 |
Tool-calling is a two-turn dependent chain with a distractor tool that must not be called — a single trivial call is too easy to discriminate between tiers.
Usage
llama-server -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-APEX-mini.gguf \
--ctx-size 131072 -fa on --jinja
The chat template is embedded, so --jinja is enough. Always pass --ctx-size — it
otherwise defaults to the model's trained context.
To use the MTP head for speculative decoding:
llama-server -m ...-APEX-i-quality.gguf --ctx-size 131072 -fa on --jinja \
--spec-type draft-mtp --spec-draft-n-max 2
Draft depth is model-specific and does not transfer between models — measure it at your own sampling settings and context length rather than copying a number from elsewhere.
This model emits explicit reasoning traces before its answer. Budget output tokens
accordingly; a short --n-predict will truncate mid-thought.
Needs a llama.cpp new enough to load the MTP head — if you see a tensor-count mismatch on load, update.
Notes on the architecture
The row dimensions are 2688 (hidden) and 1856 (expert-down), neither divisible by 256. Every
k-quant and IQ type requires a 256-wide superblock, so on this model they are all illegal and
llama-quantize silently substitutes other types. A stock Q4_K_M of this model measures
6.21 bpw against a nominal 4.85, and contains ~1% actual Q4_K. That is why these tiers use
block-32 and block-64 types throughout, chosen deliberately rather than arrived at by fallback.
Attribution
- Base model: NVIDIA — NVIDIA-Nemotron-3.5-Lightning-30B-A3B
- APEX recipe & toolkit: LocalAI — localai-org/apex-quant
- Quantization engine: llama.cpp (ggml-org)
Calibration: a general prose/scientific corpus (no code), matching the other APEX quants in this collection. Unofficial community quantization; not affiliated with or endorsed by NVIDIA.