Nemotron 3 Nano 30B — KD Distilled
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 fine-tuned with LoRA on 3,872 reasoning traces distilled from DeepSeek-V4-Flash (284B), with the adapter merged into the base weights.
Held-out pass@1 on coding problems: 23.0% → 63.4%, which is 92.5% of the teacher's own score at 9.5× fewer parameters.
Corpus construction, training, merging, and evaluation all ran on a single HP ZGX Fury (one NVIDIA GB300 Grace Blackwell Ultra). No cloud APIs at any stage.
Results
213 held-out problems (182 APPS, 8 HumanEval+), unseen during training. Greedy decoding.
| base | this model | teacher (284B) | |
|---|---|---|---|
| pass@1 | 23.0% | 63.4% | 68.5% |
| APPS (n=182) | 22.5% | 69.8% | 75.8% |
| hard (n=26) | 3.8% | 65.4% | 73.1% |
| easy (n=66) | 27.3% | 56.1% | 56.1% |
| responses that terminated | 34.3% | 93.0% | 100.0% |
| mean completion tokens | 9,736 | 3,125 | 875 |
| responses with no code block | 138 | 11 | 8 |
What actually changed
The gain is about finishing, not about better algorithms.
The base model failed 138 of 213 problems by never producing code at all. It would analyse the problem correctly, describe a sound approach, and then run past the token budget mid-thought. Only 34% of its responses terminated on their own, at a mean of 9,736 tokens.
The teacher terminates every time, in 875 tokens. This model lands in between: 93% termination at 3,125 tokens. What transferred was the decision to stop analysing and commit to an answer, which matters most on hard problems, where the base model spirals longest.
On easy problems it matches the teacher exactly.
Usage
Loads and serves like the base model.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "curtburk/nemotron3-nano-kd-distilled"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True,
)
messages = [{"role": "user", "content": "Write a function returning the nth Fibonacci number."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
out = model.generate(ids, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
With vLLM:
vllm serve curtburk/nemotron3-nano-kd-distilled \
--trust-remote-code --max-model-len 16384 --gpu-memory-utilization 0.85
Requires mamba-ssm and causal-conv1d, same as the base model. Generating with
transformers directly may warn about NemotronHHybridDynamicCache; pass
use_cache=False or use vLLM.
Training
| method | LoRA, merged into base weights |
| framework | Megatron-Bridge (nvcr.io/nvidia/nemo:25.11.nemotron_3_nano) |
| rank / alpha | 32 / 32 |
| target modules | linear_qkv, linear_proj, linear_fc1, linear_fc2, in_proj, out_proj |
| learning rate | 1e-4, cosine, 75 warmup steps |
| sequence length | 4,096, unpacked |
| batch | micro 1, global 8 |
| iterations | 1,500 (checkpoint taken at 1,400) |
| precision | bf16 |
| hardware | 1× NVIDIA GB300, ~2 hours at 5.2 s/step |
| loss | 0.73 → 0.46, no NaN iterations |
Merged adapters cover attention, Mamba in_proj/out_proj, and MoE shared experts. Merging the routed expert adapters as well was tested and scored 8 points lower (55.4% vs 63.4%) — the model more often produced a complete analysis and then stopped without writing a solution. See the repo for details.
Note: Megatron-Bridge's save_hf_pretrained() does not merge subclass-style LoRA
adapters and fails silently, producing an export byte-identical to the base model. The
merge here was done in-place before export. See the repo for details and a working
script.
Caveats
Single run. One seed, one checkpoint, one benchmark family. The direction held across every slice measured, but treat the exact magnitude as approximate.
Teacher score is a contaminated ceiling. The teacher generated traces for these same 213 problems during corpus construction, so the split is held out from the student but not from the teacher. Retention is approximate. The teacher also ran at a smaller token budget (10,240 vs 12,288) and on a different vLLM build.
Sampling mismatch. Evaluation used greedy decoding; the corpus was generated at
temperature=1.0, top_p=1.0, which is what the teacher's model card recommends.
MBPP excluded. MBPP scored 1/23 for every model tested including the 284B teacher, which is a harness bug rather than a model property. It is left out of the table above rather than reported as a real number.
Format adherence did not transfer. The corpus uses a fixed markdown structure (Analysis / Approach / Implementation Plan / Solution / Verification). This model reproduces the full structure on only 2.3% of responses. It learned termination behaviour, not formatting.
Not evaluated outside coding. No measurement of whether general capability, chat quality, or long-context behaviour regressed. Assume they may have.
Licenses
Base model under the NVIDIA Open Model License. Training traces generated from DeepSeek-V4-Flash, which is MIT-licensed.
Links
- Training corpus: curtburk/nemotron3-nano-kd-corpus
- Pipeline and evaluation harness: https://github.com/curtburk/distillation-nemotron3