Composite GGUF artifact combining the unsloth UD-Q4_K_XL quantization of NVIDIA Nemotron-3-Super-120B-A12B with NVIDIA's official MTPv2 multi-token prediction head, spliced in as a native NextN draft block for speculative decoding with Mesh LLM.
Model Overview
| Property | Value |
|---|---|
| Source target | unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF (UD-Q4_K_XL, 3 shards) |
| Source draft | nvidia/Nemotron-3-Super-120B-A12B-BF16-MTPv2 (BF16 MTP head) |
| Base model | nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 |
| Family | Nemotron 3 (hybrid SSM + latent MoE) |
| GGUF architecture | nemotron_h_moe |
| Quantization | UD-Q4_K_XL (target) + BF16 (draft block) |
| Layers | 88 target + 1 MTP NextN block (blk.88) |
| MTP metadata | nemotron_h_moe.nextn_predict_layers=1 |
| Context length | 1,048,576 |
| Artifact size | ~84 GiB across 3 shards |
What Is Included
| Artifact | Size | Contents |
|---|---|---|
...-00001-of-00003.gguf | 7 MB | Metadata shard (rewritten: block_count 88→89, nextn_predict_layers=1) |
...-00002-of-00003.gguf | 46.5 GiB | Middle shard, byte-identical to the source |
...-00003-of-00003.gguf | 37.0 GiB | Last shard recomposed with the BF16 MTP draft tensors appended |
How It Was Built
The target's quantized shards are kept untouched wherever possible; only shard 1
(metadata) and shard 3 (draft append) are rewritten. NVIDIA's BF16 MTPv2 head is
converted with the pinned llama.cpp converter (convert_hf_to_gguf.py --mtp),
which folds the two MTP layers into a single blk.88 NextN block, then spliced
past the target tensors with skippy-quantize compose-mtp (byte-copied
verbatim).
- Pipeline:
skippy-quantizecompose-mtp + validate-mtp-attach - mesh-llm revision:
bbd84c238311562f968bbbcaf3a8b2727a6ac642(PR #1439) - Build job: meshllm/6a8ed10345686a1580c0a52b
Validation
validate-mtp-attach passed on the published composite in the build job: the
3-part model loads and the MTP draft attaches natively (parts=3, layers=89,
mtp_layers=1, session created).
Recommended Use
- Local and private inference with Mesh LLM, using Skippy split serving across machines (the ~84 GiB artifact is designed for sharded serving — see docs/SKIPPY_SPLITS.md).
- Native Nemotron MTPv2 speculative decoding: one ordinary next token plus one MTP-predicted token per decode step, verified against the target.
- OpenAI-compatible chat/completions workflows through Mesh LLM's local API.
Quickstart
# Download the composite (all 3 shards).
hf download meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF \
--include '*-MTPv2-0000*-of-00003.gguf'
# Run locally with Mesh LLM (split serving across your machines).
mesh-llm serve --model "meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF"
# Check the local OpenAI-compatible model list.
curl -s http://localhost:3131/v1/models
Why MTP Matters
MTP (multi-token prediction) lets the model produce a next token and an extra predicted token in the same decode step. The runtime uses the extra token as a speculative candidate and verifies it against the target instead of waiting for a full model pass per token — higher throughput at the same quality, since every accepted candidate is verified by the target itself.
License
This artifact derives from NVIDIA's Nemotron-3-Super-120B-A12B weights and unsloth's UD-Q4_K_XL quantization of them. License terms follow the source repositories — see nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 and unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF. For architecture details, chat template guidance, and sampling recommendations, see those model cards.