Ornith-1.5-397B-MXFP4
MXFP4-quantized build of ornith-ai/Ornith-1.5-397B — the same model at roughly half the disk and memory footprint, targeting single-GPU serving on 288 GB AMD Instinct MI355X.
- Checkpoint size: ~239 GB
- Quantization: OCP MXFP4, block size 32
- Quantizer: AMD Quark 0.12.post1
- GPU tested: 1× AMD Instinct MI355X, 288 GB HBM
- Tensor parallelism: TP=1
- Architecture: Qwen3.5 MoE / Ornith 1.5
- Routed MoE expert weights are MXFP4; attention, shared experts, gates, MTP, visual components and
lm_headremain BF16.
Recommended serving
The checkpoint is tested with SGLang ROCm 7.2.4 on MI355X. AITER and native NEXTN/MTP are supported.
docker run --rm \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
--security-opt seccomp=unconfined \
--ipc=host \
--shm-size=32g \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e SGLANG_USE_AITER=1 \
-e SGLANG_USE_AITER_UNIFIED_ATTN=1 \
-e AITER_FLYDSL_FORCE=1 \
-e SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
lmsysorg/sglang-rocm:v0.5.18-rocm724-mi35x-20260830 \
python3 -m sglang.launch_server \
--model-path truespirit/Ornith-1.5-397B-MXFP4 \
--tp 1 \
--attention-backend aiter \
--page-size 16 \
--reasoning-parser deepseek-r1 \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--mem-fraction-static 0.95 \
--max-running-requests 4 \
--context-length 81920 \
--host 0.0.0.0 \
--port 8000
The server exposes an OpenAI-compatible API.
Measured on 1× MI355X
With the configuration above:
- ~180–200 output tok/s single-stream decode
- ~290 output tok/s aggregate at concurrency 4
- ~0.1 s TTFT on the tested 16K-prompt workload
- ~86K tokens measured single-request input ceiling
- two concurrent ~62K-token inputs verified
Results are serving benchmarks for this specific hardware/runtime configuration and are not model-quality benchmarks.
For general reasoning, temperature=0.6, top_p=0.95 is a useful starting point. For coding/agent workloads, temperature=1.0, top_p=0.95 is also suitable.
Quantization
Only routed MoE expert weights (*.mlp.experts.*) were quantized to MXFP4. Components kept at higher precision follow the AMD Qwen3.5 MXFP4 recipe.
Quantization can cause numerical differences from the source FP8 checkpoint.
License
MIT, inherited from the base model. See the original ornith-ai/Ornith-1.5-397B model card for model details, intended use and limitations.