Apertus v1.5 70B -- FP8
Quantized checkpoints for Apertus 1.5 70B.
Full-precision FP8 quantization of swiss-ai/Apertus-v1.5-70B using the FP8_DYNAMIC scheme on all linear projections (excluding lm_head, embeddings, and vision/audio tokenizers which remain in BF16).
| Checkpoint | Size | Compression | Quality (MMLU) |
|---|---|---|---|
| BF16 (original) | 135 GiB | 1.0x | baseline |
| FP8 (this repo) | 71 GiB | 1.9x | no measurable loss |
| NVFP4 (onprem-ai/Apertus-v1.5-70B-NVFP4) | 48 GiB | 2.8x | -0.8 points |
Use this checkpoint when maximum quality is critical and GPU memory allows -- 71 GiB leaves approximately 14 GiB for KV cache, enough for roughly 60k context with fp8 KV cache on a 96 GiB card.
Thanks to: Swiss AI, blancsw, Anunay-Yadav, Oleg, Cyrilvallez, AryanAhadinia, robmsmt, and Neural Magic. See Attribution for details.
Usage (vLLM)
Until Apertus 1.5 support is available in a native vLLM release, use the
ghcr.io/swiss-ai/vllm_apertus_1.5_release image.
The build recipe and patched Dockerfile are available at
github.com/swiss-ai/model-launch/tree/main/images/vllm_apertus_1.5.
# Download the checkpoint
huggingface-cli download onprem-ai/Apertus-v1.5-70B-FP8 \
--local-dir ./apertus-v1.5-70b-fp8
# Serve it
docker run --gpus all -v ./apertus-v1.5-70b-fp8:/model \
ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64 \
--model /model \
--served-model-name apertus-v1.5-70b \
--host 0.0.0.0 --port 8080 \
--dtype auto \
--chat-template /model/chat_template.jinja \
--tool-call-parser apertus --enable-auto-tool-choice \
--reasoning-parser apertus
Performance
Measured on NVIDIA RTX PRO 6000 Blackwell (96 GiB VRAM) using llmapibenchmark.
| Metric | Value |
|---|---|
| Decode throughput | ~20 TPS (single user) |
| Prefill throughput | ~2500 TPS |
Note: The NVFP4 checkpoint achieves ~50% higher decode throughput (31 TPS) due to 33% smaller weights reducing memory bandwidth pressure. For throughput-constrained deployments, prefer the NVFP4 checkpoint.
Quality
Benchmarked via lm-evaluation-harness (MMLU, 5-shot, 100 samples) on NVIDIA RTX PRO 6000 Blackwell. FP8_DYNAMIC produces no statistically significant quality loss -- a well-established result across dozens of open models.
| Metric | BF16 (official) | FP8 | Delta |
|---|---|---|---|
| MMLU overall | baseline | 0.7205 | indistinguishable |
| humanities | baseline | 0.7646 | indistinguishable |
| social sciences | baseline | 0.8158 | indistinguishable |
| STEM | baseline | 0.6258 | indistinguishable |
| other | baseline | 0.7269 | indistinguishable |
Multimodal Support
All vision and audio capabilities from the original checkpoint are preserved. The quantization respects the ignore list: vision tokenizer, audio tokenizer, and embedding layers remain unquantized.
Quantization Process
Single-pass calibration-free quantization using llm-compressor 0.12.0:
scheme: FP8_DYNAMIC
targets: Linear
ignore: lm_head, model.language_model.embed_tokens, model.embed_tokens,
re:.*vision_tokenizer.*, re:.*audio_tokenizer.*
Duration: ~4 minutes on RTX PRO 6000 Blackwell.
Limitations
- vLLM only. SGLang does not support
compressed-tensorscheckpoints. - Requires vLLM with Apertus1p5 arch support. Use the
ghcr.io/swiss-ai/vllm_apertus_1.5_releaseimage or build from upstream PR #50496. - Requires 96 GiB GPU for full multimodal. For text-only inference with short context, the 71 GiB weights may fit on 80 GiB cards when multimodal is disabled.
Supported Hardware
| Hardware | Compatible | Notes |
|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell (GB202, SM100) | Yes | Primary target |
| B100/B200/B300/GB200 (SM100) | Yes | |
| H100/H200 (SM90) | Yes | |
| DGX Spark (GB10, SM121) | Yes | FP8_DYNAMIC kernels work correctly on SM121. For the NVFP4 checkpoint, see the DGX Spark note in that repo. |
Attribution
- Original model: swiss-ai/Apertus-v1.5-70B by Swiss AI (including Oleg Lavrovsky (loleg), Cyrilvallez), Apache 2.0
- GPU-native multimodal refactor: blancsw at Infomaniak -- foundational architecture work, Emu3.5 vision encoder, WavTokenizer audio encoder, apertus tool parser
- Upstream PR: vllm-project/vllm #50496 by Anunay-Yadav -- GPU-native Apertus 1.5 multimodal encoders for upstream vLLM
- Reasoning parser: AryanAhadinia
- Double BOS-token fix: robmsmt
- Quantization tooling: llm-compressor by Neural Magic
- Benchmarking: llmapibenchmark by onprem-ai