Ornith-1.0-35B-FP8-BLOCK-MTP

This is the official MTP-enabled derivative of shisa-ai/Ornith-1.0-35B-FP8-BLOCK. It keeps the original FP8_BLOCK / compressed-tensors Ornith base weights and adds a BF16 Qwen3.6 MTP head in model-mtp.safetensors.

This artifact was built internally as a zero-training Qwen3.6 MTP graft, but the public upload name is: shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP. Based on our matched local vLLM tests, this is the recommended Ornith 35B MTP checkpoint for throughput. The best measured row used num_speculative_tokens=3 and reached 246.522 output tok/s on our validation workload, a +21.4% improvement over the no-spec baseline.

The MTP recipe was adapted from protoLabsAI/Ornith-1.0-9B-MTP, which demonstrated grafting a same-family Qwen MTP head into an Ornith checkpoint and optionally KL-distilling only the MTP head. Thanks to protoLabsAI for the MTP graft/distillation technique and base recipe.

What Changed

• Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
• Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
• Added shard: model-mtp.safetensors
• Added tensors: 19 top-level mtp.* tensors
• MTP dtype: BF16
• Base weights: unchanged FP8_BLOCK / compressed-tensors
• Training: none; this is a direct MTP tensor graft

No private training corpus is needed for this checkpoint because it is not distilled. The graft copies the donor mtp.* tensors into the Ornith FP8_BLOCK checkpoint and updates the safetensors index.

Important Result Summary

This is the strongest Ornith 35B MTP checkpoint from our local serving tests. It also beat the one-epoch Qwen3.6 KL-distilled derivative in matched vLLM serving, even though that distilled derivative had a slightly better offline KL proxy.

Recommended starting point:

• Use num_speculative_tokens=3 when optimizing for output throughput.
• Use num_speculative_tokens=1 or no speculative decoding if your workload is more sensitive to inter-token latency or acceptance stability.
• Re-benchmark on your own traffic before making it a production default.

Local Benchmark Methodology

Hardware:

• One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• Single-GPU serving on GPU0 only

Runtime:

• vLLM 0.23.0
• FlashInfer attention backend
• FP8 KV cache
• compressed-tensors quantization
• No LMCache for the benchmark rows below
• MAX_MODEL_LEN=32768
• MAX_NUM_SEQS=16
• MAX_BATCHED_TOKENS=32768
• MAX_CUDAGRAPH_CAPTURE_SIZE=16
• GPU_MEMORY_UTIL=0.95

Workload:

• Private custom validation benchmark derived from local code/agentic prompts
• 64 requests
• 63,327 total input tokens
• 16,384 generated tokens (256 per request)
• max_concurrency=1
• request_rate=inf
• temperature=0.6, top_p=0.95, top_k=20
• ignore_eos

The private benchmark data is not uploaded. The benchmark used vLLM's bench serve --dataset-name custom path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with prompt and output_tokens fields and run a command like:

CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp \
  --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Comparison against the companion KL-distilled artifact:

The distill checkpoint is planned as shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill. It improved the offline proxy, but it did not beat this official MTP checkpoint in matched serving throughput.

Offline proxy on the same validation split:

• Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
• Qwen3.6 KL-distill: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Internal Alternatives Tested

We also tested a Qwen3.5 donor graft, but it is not planned for upload. The Qwen3.5 rows were from an earlier MTP1-only ShareGPT sweep, so they are useful as donor-selection context rather than a direct replacement for the validation table above.

The Qwen3.5 graft was marginally faster in those MTP1 rows, but it had lower acceptance. The later Qwen3.6 validation sweep at MTP3 produced the best retained throughput result, so Qwen3.6 is the donor used for the official upload.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --served-model-name ornith-35b-fp8-block-mtp \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

num_speculative_tokens=3 gave the best local output throughput for this checkpoint, but it also reduced acceptance to about 67% and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

• Source model: deepreinforce-ai/Ornith-1.0-35B
• Quantization tool: llm-compressor model-free PTQ
• Quantization format: compressed-tensors
• Scheme: FP8_BLOCK
• Calibration data: none; this is data-free/model-free PTQ
• Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
• Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
• Target modules: Linear
• compressed-tensors metadata version recorded in config.json: 0.15.1.a20260406

The quantization ignore list includes re:^mtp.*, so the grafted MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

• Source model: deepreinforce-ai/Ornith-1.0-35B
• FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
• MTP donor: Qwen/Qwen3.6-35B-A3B
• MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}