Model Overview
- Model Architecture: NemotronHForCausalLM
- Input: Text
- Output: Text
- Total Parameters: 550B
- Active Parameters: 55B
- Model Optimizations:
- Weight quantization: FP8 (per-channel)
- Activation quantization: FP8 (dynamic per-token)
- Intended Use Cases:
- Reasoning and complex problem solving.
- Mathematics and science.
- Code generation.
- Instruction following.
- Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
- Release Date: 06/04/2025
- Version: 1.0
- Model Developers: Red Hat
Quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
Model Optimizations
This model was obtained by quantizing the weights and activations of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators within transformer blocks are quantized. Weights are quantized with FP8 per-block quantization, while activations are quantized with FP8 dynamic per-token quantization. The llm-compressor library is used for quantization.
Deployment
Use with vLLM
This model can be deployed efficiently using the vLLM backend.
Install dependencies:
uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install llmcompressor
Launch the vLLM server:
vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block \
--host 0.0.0.0 --port 8088 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 32 \
--max-num-batched-tokens 32768 \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser nemotron_v3 \
--mamba-ssm-cache-dtype float16 \
--mamba-backend flashinfer \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
--trust-remote-code
Send requests:
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8088/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
model = "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block"
messages = [
{"role": "user", "content": "Solve for x: 2x + 5 = 13"},
]
outputs = client.chat.completions.create(
model=model,
messages=messages,
)
generated_text = outputs.choices[0].message.content
print(generated_text)
Creation
This model was quantized using the llm-compressor library as shown below.
from llmcompressor import model_free_ptq
MODEL_ID = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-block"
model_free_ptq(
model_stub=MODEL_ID,
save_directory=SAVE_DIR,
scheme="FP8_BLOCK",
ignore=[
"re:.*gate$",
"lm_head",
"model.embed_tokens",
"re:.*mixer.conv1d.*",
"re:.*norm_f*",
"re:.*bias$",
"re:.*embed_tokens$",
"backbone.embeddings"
],
max_workers=15,
device="cuda:0",
)
Evaluation
The model was evaluated on reasoning tasks using lighteval. vLLM was used as the serving backend for all evaluations.
Install dependencies:
uv pip install git+https://github.com/vllm-project/vllm.git
uv pip install lighteval==0.13.0
uv pip install "litellm[caching]>=1.66.0"
Launch the vLLM server:
vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block \
--host 0.0.0.0 --port 8088 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 262144 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 32 \
--max-num-batched-tokens 32768 \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser nemotron_v3 \
--mamba-ssm-cache-dtype float16 \
--mamba-backend flashinfer \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
--model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}' \
--trust-remote-code
AIME 2025:
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
"aime25|0" \
--output-dir results --save-details
GPQA Diamond:
lighteval endpoint litellm \
"model_name=hosted_vllm/RedHatAI__NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block,provider=hosted_vllm,base_url=http://127.0.0.1:8088/v1,timeout=3600,concurrent_requests=32,generation_parameters={temperature:1.0,top_p:0.95,max_new_tokens:32768}" \
"gpqa:diamond|0" \
--output-dir results --save-details
Accuracy
Extended Evaluation
The model was evaluated across instruct, reasoning, and coding tasks. Scores are averaged over multiple seeds (3 seeds for most benchmarks, 8 for AIME 2025). Recovery is computed as the ratio of the quantized model score to the base model score.
Tool Calling Evaluation
The model was evaluated on tool calling tasks using the Berkeley Function-Calling Leaderboard v4 (BFCLv4). vLLM was used as the serving backend for all evaluations.