mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit
Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs
NVIDIA's Mamba-attention hybrid MoE, running locally on a Mac. This is an OptiQ mixed-precision quant of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. 22.8 GB on disk, down from 65.8 GB.
Lightning is a nemotron_h hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from.
What it is
| Property | Value |
|---|---|
| Base | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (~30B total, ~3B active) |
| Method | OptiQ mixed-precision, sensitivity-driven (uniform-4-bit reference) |
| Precision | per-layer 4/8-bit: 36 layers at 4-bit, 127 at 8-bit |
| Experts | 46 fused expert tensors: 36 at 4-bit, 10 at 8-bit |
| On disk | 22.8 GB |
| Speculative decoding | MTP head preserved in optiq/mtp.safetensors |
All 164 quantizable projections were measured. Every 4-bit assignment in the model is an expert tensor: the 128-expert stack is where the size is, and the sweep found it is also where the precision is least needed.
Following llama.cpp's naming for mixed quants, the "4bit" label denotes the family, not the weighted average.
Capability Score
Six-metric mean (the standard OptiQ text eval). Strong code generation; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers would predict.
| Metric | Score |
|---|---|
| MMLU (5-shot, 969 samples) | 78.9% |
| GSM8K (1000 samples) | 85.5% |
| IFEval (full set, strict) | 68.4% |
| BFCL-V3 simple (200 calls) | 83.0% |
| HumanEval (164 problems, pass@1) | 87.8% |
| HashHop (long-context retrieval) | 40.0% |
| Capability Score (mean of 6) | 73.94 |
Run it
pip install "mlx-optiq>=0.4.20"
import optiq # registers the nemotron_h MoE + MTP sidecar
from mlx_lm import load, generate
model, tok = load("mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit")
msgs = [{"role": "user", "content": "Explain why the sky is blue."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=600))
It is a reasoning model and emits a <think> block before its answer, so give it room and read the text after </think>.
Serve it on a smaller Mac
At 22.8 GB this does not fit resident on a 24 GB machine. It is a 128-expert MoE, so stream the experts from SSD instead:
optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts
Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory. The same policy is available from the library API via OptiqEngine(path, stream_experts="auto"), and from optiq eval --stream-experts for benchmarking.
Links
- Project website: mlx-optiq.com
- Family guide: mlx-optiq.com/docs/nemotron3
- All OptiQ quants: mlx-optiq.com/models
- Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16