NVIDIA-Nemotron-3.5-Lightning-30B-A3B - GGUF

This repository contains GGUF format model files for NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

These files were converted and quantized using llama.cpp.

Available Files

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The pre-training data has a cutoff date of September 2025.
• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

Description

NVIDIA-Nemotron-3.5-Lightning-30B-A3B is a large language model (LLM) trained by NVIDIA.

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside speculative decoding methods (DSpark, DFlash, MTP) for faster text generation. The model has 3B active parameters and 30B parameters in total.

This model is ready for commercial use under the OpenMDW-1.1 license.

Usage with llama.cpp

CLI / llama-cli

Reasoning ON (default):

llama-cli \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  --jinja \
  --chat-template-file chat_template.jinja \
  -p "Write a Python function to compute Fibonacci numbers." \
  --temp 1.0 --top-p 0.95 \
  -ngl 99

llama-server

Start the OpenAI-compatible server:

llama-server \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  --temp 1.0 --top-p 0.95 \
  -np 1 \
  -c 40960 \
  --port 8000 \
  -ngl 99 \
  -fa on \
  --jinja \
  --chat-template-file chat_template.jinja \
  --no-webui \
  --fit off

With DFlash Speculative Decoding

Accelerate token generation using the DFlash draft model:

llama-server \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  -md nemotron-3.5-lightning-30b-a3b-dflash.gguf \
  --draft-max 6 \
  --temp 1.0 --top-p 0.95 \
  -np 1 \
  -c 40960 \
  --port 8000 \
  -ngl 99 \
  -ngld 99 \
  -fa on \
  --jinja \
  --chat-template-file chat_template.jinja \
  --no-webui \
  --fit off

API Client Example (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# Reasoning ON (default)
response = client.chat.completions.create(
    model="nemotron-3.5-lightning-30b-a3b",
    messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
    max_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)

# Reasoning OFF (direct answer)
response = client.chat.completions.create(
    model="nemotron-3.5-lightning-30b-a3b",
    messages=[{"role": "user", "content": "What is the capital of Japan?"}],
    max_tokens=128,
    temperature=1.0,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Benchmarks

Reasoning Benchmark Evaluations

License and Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.