Nemotron-3-Nano-Omni-30B-A3B-Reasoning · JANG_6M
29.1 GB · ~110.0 tok/s decode on M5 Max · 31B total / ~3B active · hybrid Mamba-2 + Attention + MoE · native MLX
Full multimodal (text + image + audio + video) port of NVIDIA's Nemotron-3-Nano-Omni-30B-A3B-Reasoning to Apple MLX — calibrated with a measured Hessian, an imatrix fit and AWQ, with the RADIO vision tower and the Parakeet speech encoder kept at fp16 so every modality actually works.
| Modality | Status |
|---|---|
| Text | ✅ reasoning on by default |
| Image (RADIO ViT, 512px tiles + thumbnail) | ✅ |
| Video (frame sampling + EVS pruning) | ✅ |
| Audio / ASR (Parakeet 24-layer Conformer) | ✅ |
| Tool calling (agentic) | ✅ XML <tool_call><function=…> |
Quantization recipe
JANG_6M is a mixed-precision allocation (6-bit, 8-bit), not a single width.
The bit map is chosen per tensor role, then calibrated:
| Component | Precision |
|---|---|
Routed experts switch_mlp.fc1 / fc2 (93% of params) | 6-bit / 6-bit |
Shared expert, attention q/k/v/o, lm_head | 8-bit |
Mamba-2 in_proj / out_proj, embeddings | 8-bit |
conv1d, A_log, D, dt_bias, router gate, all norms | fp16 |
| Vision tower, Parakeet encoder, both projectors | fp16 (never quantized) |
Group size 64. 164 quantized modules; 1106 tower tensors passed through untouched.
Why there is no 2-bit variant
This model's expert MLP is unusually sensitive to low-bit quantization, so the routed experts carry a hard minimum width regardless of profile. A nominal "2-bit" build lands within ~7% of the 4-bit one and is measurably worse — not a 2-bit bundle in any useful sense, so we do not ship one.
Calibration
Quantization is calibrated against measured activations from a domain-weighted corpus (coding, agentic/tool, reasoning, general, multilingual, long-context), not fitted to the weights alone:
- imatrix — the affine fit is weighted toward the channels that actually carry signal.
- AWQ — per-channel scales, folded so the result is numerically equivalent and costs nothing at inference.
- Hessian — captured per MoE layer and used to steer the bit allocation.
QAT is not available for this model: NVIDIA's own quantized releases are
post-training (modelopt, MIXED_PRECISION), and NVFP4 codes carry no meaning
in an affine (scale, bias) layout.
Sampling — use the right preset
Reasoning is ON by default (enable_thinking); turning it off prefills an
empty <think></think> block rather than omitting it.
| Mode | temperature | top_p | top_k | max_tokens | reasoning_budget | grace |
|---|---|---|---|---|---|---|
| Thinking (default) | 0.6 | 0.95 | — | 20480 | 16384 | 1024 |
| Instruct (non-thinking) | 0.2 | — | 1 | 1024 | — | — |
| ASR / transcription | 1.0 | — | 1 | — | — | — |
Stamped into both generation_config.json and
jang_config.chat.sampling_defaults. EOS is [2, 11] — dropping 11 means the
model never stops.
All bundles in this family
| Variant | Size | Notes |
|---|---|---|
| JANG_4M | — | |
| JANG_6M | 29.1 GB | this bundle |
| MXFP4 | — |
Install
hf download OsaurusAI/Nemotron-3-Nano-Omni-30B-A3B-JANG_6M \
--local-dir Nemotron-3-Nano-Omni-30B-A3B-JANG_6M
Runs in Osaurus / vMLX, which carries the
native MLX NemotronHOmni stack (RADIO vision, Parakeet audio, EVS video). The
LLM half also loads directly with mlx_lm:
from mlx_lm import load, generate
model, tokenizer = load("Nemotron-3-Nano-Omni-30B-A3B-JANG_6M")
Credits
Quantized by Jinho Jang (eric@osaurus.ai). Base model © NVIDIA, released under the NVIDIA Open Model License.