NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is the full-precision reference release of NVIDIA's Nemotron 3.5 Lightning model, a hybrid Mixture-of-Experts architecture interleaving Mamba-2 and MoE layers with select Attention layers, totaling 30B parameters with only 3B active at inference and supporting up to 1M tokens of context (256K on a single H100). Pre-trained on over 20 trillion tokens using an NVFP4 recipe and enhanced with Multi-Token Prediction (MTP) layers for richer training signals, it underwent a four-stage pipeline — pretraining, MTP continued pretraining, supervised fine-tuning on code/math/science/tool-calling data, and multi-environment GRPO reinforcement learning across math, code, instruction-following, and structured-output tasks — supporting configurable reasoning mode (
enable_thinking), English plus five additional languages, and speculative decoding via DSpark, DFlash, or MTP for faster generation. This BF16 checkpoint is intended primarily as a starting point for customization — post-training (SFT/RL/distillation), domain adaptation, or producing quantized (NVFP4, W4A16, GGUF) variants — rather than direct production deployment, for which NVIDIA recommends the companion NVFP4 release; on NVIDIA's internal benchmark harness it trails larger sibling Qwen3.6-35B-A3B and Nemotron 3 Super on most agentic coding and reasoning tasks (e.g., 51.56 on SWE-bench Verified, 75.44 on GPQA Diamond) while remaining competitive with Gemma4-26B-A4B and GPT-OSS-20B, deployable on single or multi-GPU H100/H200/GB200/A100 setups via vLLM, and released under the OpenMDW-1.1 license alongside NVIDIA's full pre-training and post-training dataset lineage.
Model Files
| File Name | Quant Type | File Size | File Link |
|---|---|---|---|
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.F16.gguf | F16 | 65.9 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf | Q4_K_M | 25.4 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q5_K_M.gguf | Q5_K_M | 27 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q8_0.gguf | Q8_0 | 35 GB | Download |
Quick Start with llama.cpp
FROM ghcr.io/ggml-org/llama.cpp:full
WORKDIR /app
RUN apt update && apt install -y python3-pip
RUN pip install -U huggingface_hub --break-system-packages
RUN python3 -c 'from huggingface_hub import hf_hub_download; \
repo="prithivMLmods/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF"; \
hf_hub_download(repo_id=repo, filename="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", local_dir="/app")'
CMD ["--server", \
"-m", "/app/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", \
"--host", "0.0.0.0", \
"--port", "7860", \
"-t", "2", \
"--cache-type-k", "q8_0", \
"--cache-type-v", "iq4_nl", \
"-c", "128000", \
"-n", "38912"]
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp