Model Overview
- Model Architecture: NemotronHForCausalLM
- Input: Text
- Output: Text
- Total Parameters: 550B
- Active Parameters: 55B
- Model Optimizations:
- Activation quantization: FP8
- Weight quantization: FP8
- Intended Use Cases:
- Reasoning and complex problem solving.
- Mathematics and science.
- Code generation.
- Instruction following.
- Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
- Release Date: 06/04/2025
- Version: 1.0
- Model Developers: Red Hat
Quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
Model Optimizations
This model was obtained by quantizing the weights and activations of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.
Only the weights and activations of the linear operators within transformer blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.
Deployment
Use with vLLM
This model can be deployed efficiently using the vLLM backend.
Install dependencies:
uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install llmcompressor
Launch the vLLM server:
vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
--host 0.0.0.0 --port 8088 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 32 \
--max-num-batched-tokens 32768 \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser nemotron_v3 \
--mamba-ssm-cache-dtype float16 \
--mamba-backend flashinfer \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
--trust-remote-code
Send requests:
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8088/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
model = "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic"
messages = [
{"role": "user", "content": "Solve for x: 2x + 5 = 13"},
]
outputs = client.chat.completions.create(
model=model,
messages=messages,
)
generated_text = outputs.choices[0].message.content
print(generated_text)
Creation
This model was quantized using the llm-compressor library as shown below.
from llmcompressor import model_free_ptq
MODEL_ID = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"
model_free_ptq(
model_stub=MODEL_ID,
save_directory=SAVE_DIR,
scheme="FP8_DYNAMIC",
ignore=[
"re:.*gate$",
"lm_head",
"model.embed_tokens",
"re:.*mixer.conv1d.*",
"re:.*norm_f*",
"re:.*bias$",
"re:.*embed_tokens$",
"backbone.embeddings"
],
max_workers=15,
device="cuda:0",
)
Evaluation
The model was evaluated on reasoning tasks using lighteval. vLLM was used as the serving backend for all evaluations.
Install dependencies:
uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install lighteval==0.13.0
uv pip install "litellm[caching]>=1.66.0"
Launch the vLLM server:
vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
--host 0.0.0.0 --port 8088 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 32 \
--max-num-batched-tokens 32768 \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser nemotron_v3 \
--mamba-ssm-cache-dtype float16 \
--mamba-backend flashinfer \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
--trust-remote-code
AIME 2025:
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
"aime25|0" \
--output-dir results --save-details
GPQA Diamond:
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
"gpqa:diamond|0" \
--output-dir results --save-details
Accuracy
Multi-category Evaluation
Tool Calling Evaluation
The model was evaluated on tool calling tasks using the Berkeley Function-Calling Leaderboard v4 (BFCLv4). vLLM was used as the serving backend for all evaluations.