Nemotron 3.5 Lightning 30B-A3B — REAP-20B (general calibration)
A 20B-total / 3B-active derivative of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16,
pruned with REAP expert pruning (128 → 77 routed experts per MoE layer,
40% sparsity), calibrated on a general-purpose corpus with a slight coding
focus (general instruction 44% / coding 33% / math 22%, 538 samples
generated by the full model). MTP head stripped. No LoRA included — this is
the pre-LoRA base; a domain LoRA (subagent orchestration + coding/tool-calling
mix) is the intended next step.
| Base model | NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 |
| Params before / after | 31.58B → 19.87B (active stays ~3B) |
| Pruning | REAP, 40% expert sparsity, 128→77 experts/layer × 23 MoE layers |
| Calibration data | 538 samples: instruction 238, coding 180, math 120 (from the owner's general/coding/math prompt banks) |
| Quantization | None in this repo (BF16); IQ4_NL GGUF ≈ 11.5GB (see below) |
| License | OpenMDW-1.1 (same as base) |
This repository ships:
- the REAPed model (weights + config,
model-*.safetensors) - the methodology + recalibration kit (
reap/): how the pruning was calibrated, the prompt-bank mix, and how to redo it on your own data.
Why prune
Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + Attention model with
hidden=2688, moe_intermediate=1856. Neither is divisible by 256, so
llama.cpp K/IQ quant formats (256-element superblocks) cannot represent the
expert weights — every GGUF quant falls back to block-32 formats (~4.5 bpw
floor) regardless of requested bit width. At 30B that floor is ~18GB; at 20B
it is ~11.5GB. REAP removes the redundant experts so the model fits smaller
quantization budgets without touching the active computation:
- ~37% smaller total memory at the same speed (3B active unchanged)
- KV cache stays ~6KB/token (only 6 attention layers, 2 KV heads), so long contexts stay cheap on consumer hardware
- A Q4_0/IQ4_NL GGUF ≈ 11.5GB fits a 24GB MacBook with room for 32K+ context (the block-32 floor is why IQ3-class GGUF quants are not achievable for this architecture at any size)
Calibration data (the important part)
REAP prunes the experts your calibration data doesn't use. This model was calibrated on a general-purpose mix so the surviving experts serve general use:
| Source | Share | Content |
|---|---|---|
| instruction (openhermes-style) | 44% | general QA, writing, reasoning, how-to |
| coding (deepseek trajectories + opencode) | 33% | coding problems, agentic code tasks |
| math (metamath-style) | 22% | word problems, arithmetic, reasoning |
Completions were generated by the full model (NVFP4, natural thinking,
per-domain token budgets). If your workload is heavily domain-specific,
re-calibrate on your own data — see reap/.
Using this repo
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
torch_dtype=torch.bfloat16, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")
For llama.cpp / LM Studio on Apple Silicon, quantize to IQ4_NL (~11.5GB):
llama-quantize model.gguf out-IQ4_NL.gguf IQ4_NL
llama-server -m out-IQ4_NL.gguf -c 131072 --reasoning-parser nemotron_v3
Context length: -c 131072 (128K) is a safe default for 16GB-VRAM GPUs and
24GB Macs — KV is only ~6KB/token (6 attention layers, 2 KV heads), so 128K
context costs roughly 0.8GB. The model itself supports up to 1M tokens
(256K is NVIDIA's single-GPU deployment length); on laptops the practical
limit at longer contexts is prefill time, not memory — raise -c (e.g.
262144) if your workload needs it.
Validation (limited — alpha)
Sanity-checked with greedy generation on general QA, coding, and math prompts: coherent analysis, correct answers (e.g. math arithmetic verified), no repetition/looping. Full benchmark suites were not run. The closest published pruning analogue (Qwen3-30B-A3B, 128 experts, 8 active) retained ~99.8% of baseline at 50% sparsity.
Sources & how to make one yourself
- Base model: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- Algorithm: REAP — Router-weighted Expert Activation Pruning (Cerebras; implemented in vllm-project/llm-compressor)
- Pruning tooling: llm-compressor's
REAPPruningModifier, plus 3 small patches for the NemotronH architecture (documented inreap/README.md) - Calibration data: 538 samples generated by the full model from general instruction, coding, and math prompt banks (openhermes-style instruction data, DeepSeek coding-trajectory prompts, metamath-style word problems) — see
reap/generate_general_data.py - Synthetic-data guidance: NVIDIA's Build a Bash Agent with Synthetic Data and sft-data-blending patterns
- Why 20B / the quant floor: this architecture's
hidden=2688/moe_intermediate=1856are not 256-divisible, so llama.cpp GGUF quants floor at block-32 formats (~4.5 bpw); at 20B that is ~11.5GB instead of ~18GB
To reproduce on your own data: serve the full model, generate your domain corpus
with reap/generate_general_data.py (or your own loader), then run
reap/reap_nemotron.py --data yours.jsonl --sparsity 0.40. Full step-by-step,
patches, and hardware notes: reap/README.md.
Related
- LoRA adapter + IQ4_NL GGUF (recovery pass + quantized for laptops): Nemotron-3.5-Lightning-30B-A3B-REAP-20B-LoRA-IQ4NL
- Base: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
License
OpenMDW License Agreement, version 1.1
(same as the base model). See LICENSE. If you upload derived model files,
carry the same license and attribute NVIDIA for the base weights.