Nemotron-Flash-3B Instruct Model

Model Overview

Nemotron-Flash is a new hybrid small language model family designed around real-world latency rather than parameter count. It features latency-optimal depth–width ratios, hybrid operators discovered through evolutionary search, and training-time weight normalization. See our NeurIPS 2025 paper for more technical details.

The models achieve SOTA accuracy in math, coding, and commonsense reasoning at the 1B and 3B scales, while delivering decent small-batch latency and large-batch throughput. For example, Nemotron-Flash-1B achieves +5.5% accuracy, 1.9× lower latency, and 45.6× higher throughput compared with Qwen3-0.6B; and Nemotron-Flash-3B achieves +2% / +5.5% accuracy over Qwen2.5-3B / Qwen3-1.7B with 1.3× / 1.7× lower latency and 6.4× / 18.7× higher throughput, respectively.

Environment

torch<=2.9.1
transformers<=4.56.2
causal-conv1d
flash-attn<=2.7.3
mamba-ssm
flash-linear-attention

We provide a script to build the conda environment: bash setup.sh.

Chat with Nemotron-Flash

We integrated the attention kernel from TRT-LLM AutoDeploy to enable generation with CUDA Graph:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo_name = "nvidia/Nemotron-Flash-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_name, trust_remote_code=True)
model = model.cuda().to(torch.bfloat16)

print('Initializing generation state...')
generation_state = model.init_cuda_graph_generation(
    max_new_tokens=max_new_tokens,
    batch_size=1,
    device='cuda',
)

prompt = input("User:")
prompt = "User: " + prompt + "\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to('cuda')

print(f"Generating with CUDA graph acceleration...")
outputs = model.generate_with_cuda_graph(
    input_ids=inputs["input_ids"],
    generation_state=generation_state,
    max_new_tokens=256,
    temperature=0,
    eos_token_id=tokenizer.eos_token_id,
)

response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(f"Response: {response}")

Another option is to perform generation w/o CUDA Graph:

outputs = model.generate_with_cache(
    input_ids=inputs["input_ids"],
    max_new_tokens=256,
    temperature=0,
    eos_token_id=tokenizer.eos_token_id,
    )

Finetune Nemotron-Flash

To finetune Nemotron-Flash models, switch the attention kernel to FlashAttention2 when loading the model:

from transformers import AutoConfig, AutoModelForCausalLM
repo_name = "nvidia/Nemotron-Flash-3B-Instruct"

config = AutoConfig.from_pretrained(repo_name, trust_remote_code=True)
setattr(config, "attn_implementation_new", "flash_attention_2")
model = AutoModelForCausalLM.from_pretrained(repo_name, config=config, torch_dtype=torch.bfloat16, trust_remote_code=True)

Running Nemotron-Flash with TensorRT-LLM

Setup

Installation + quick start for TensorRT-LLM: Tutorial.

Quick example

An example script for running through the generation workflow:

cd examples/auto_deploy
python build_and_run_ad.py --model nvidia/Nemotron-Flash-3B-Instruct --args.yaml-extra nemotron_flash.yaml

Serving with trtllm-serve

• Spin up a trtllm server (more details are in this doc):

trtllm-serve serve nvidia/Nemotron-Flash-3B-Instruct \
--backend _autodeploy \
--trust_remote_code \
--extra_llm_api_options examples/auto_deploy/nemotron_flash.yaml

• Send a request (more details are in this doc):

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "nvidia/Nemotron-Flash-3B-Instruct",
        "messages":[{"role": "user", "content": "Where is New York?"}],
        "max_tokens": 16,
        "temperature": 0
    }'

Citation

@misc{fu2025nemotronflash,
      title={Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models}, 
      author={Yonggan Fu and Xin Dong and Shizhe Diao and Matthijs Van keirsbilck and Hanrong Ye and Wonmin Byeon and Yashaswi Karnati and Lucas Liebenwein and Hannah Zhang and Nikolaus Binder and Maksim Khadkevich and Alexander Keller and Jan Kautz and Yingyan Celine Lin and Pavlo Molchanov},
      year={2025},
      eprint={2511.18890},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2511.18890}, 
}