Ornith-1.5-35B-A3B-MTP — NInfer Artifact Card

This card documents the derived NInfer .ninfer inference artifact for the ornith-ai/Ornith-1.5-35B-A3B model, not the upstream Hugging Face repository itself.

Ornith-1.5-35B-A3B is a Qwen 3.5 Mixture-of-Experts (MoE) multimodal language model (text + image input) with a DFlash multi-token-prediction (MTP) draft head. The upstream repo publishes BF16 safetensors (17 shards, ~143.84 GB). This artifact is a single groupwise-quantized NInfer bundle built from that BF16 source, intended for serving through the NInfer engine.

This MTP variant replaces the shipped Ornith-1.5 MTP head — which was untrained (random-init, ~21–32% draft acceptance) — with a trained MTP head from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY. In our measured comparison on a prose prompt (this host, RTX 5090, single-stream CLI, spec flags from models.nix), the trained head reaches 39.0% overall draft acceptance vs 21.8% for the native head — a ~1.8× improvement and +21.5% decode throughput (336.5 vs 277.0 tok/s). See the Benchmark section.

Scope note. This is a derived inference artifact. The upstream ornith-ai/Ornith-1.5-35B-A3B repository remains the source of truth for the model weights and the canonical (MIT-licensed) distribution. What is described here is a derived, quantized, engine-specific container produced by the conversion run.

Model Architecture

The NInfer Artifact

The single self-contained inference bundle produced from the BF16 source:

Quantization scheme

The artifact uses groupwise mixed-precision integer quantization, applied on-the-fly at conversion time (Q4/Q5/Q6/W8). It is a mixed-precision layout: the four integer group-wise formats carry the bulk of the expert/MLP weights, while a set of tensors is kept in full BF16 precision.

Weight types present in the artifact:

Format histogram

(934 tensors total; the four *_F16S group-wise counts together account for the 386 quantized tensors.)

Component tensor bytes

Provenance / Conversion

This artifact was produced by the NInfer source converter (the NInfer converter source tree), running in groupwise-int4 mode (i.e. on-the-fly group-wise quantization at conversion time) using the recipe qwen3_6_35b_a3b-v2.

Converter

• Module: tools.convert.qwen3_6_35b_a3b.convert (in the NInfer converter source tree)
• Recipe: qwen3_6_35b_a3b-v2 (recipe id RECIPE_ID = "qwen3_6_35b_a3b-v2")
• Quantization mode: groupwise-int4 (Q4/Q5/Q6/W8 on the fly)
• Converter revision: a9136e412b8418a8189152e91fdd571044d0450b
• Sources:
  • Base model: ornith-ai/Ornith-1.5-35B-A3B (BF16 safetensors, 17 shards)
  • DFlash draft model: z-lab/Qwen3.6-35B-A3B-DFlash
  • Trained MTP head (graft): shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY — 15 mtp.* tensors, 897,934,336 bytes. Initialized from the Qwen/Qwen3.6-35B-A3B MoE MTP head, then re-aligned to Ornith-1.5 hidden states via full-vocabulary KL distillation (two-epoch code-heavy warm-start).

Conversion command

From the NInfer converter source tree:

PYTHONPATH=$PWD python -m tools.convert.qwen3_6_35b_a3b.convert \
  --model /home/knoopx/models/llm/ornith/ornith-ai-Ornith-1.5-35B-A3B-base \
  --dflash-model /home/knoopx/models/llm/ornith/z-lab-Qwen3.6-35B-A3B-DFlash \
  --out out/ornith-1.5-35b-a3b-mtp.ninfer --device cpu

• Device: cpu
• Elapsed: 228.1 s

Front-end Resources

The bundle embeds the model's own 6 front-end resources (its own chat_template.jinja and the other 5):

1. tokenizer.json
2. tokenizer_config.json
3. generation_config.json
4. chat_template.jinja
5. preprocessor_config.json
6. video_preprocessor_config.json

Note. Unlike the base knoopx/Ornith-1.5-35B-A3B-NInfer artifact (which embedded the base pin's chat_template.jinja), this artifact embeds the model's own chat_template.jinja — prompt formatting behaviour matches the model's native template.

Size delta. This artifact (22,783,245,824 bytes) is 256 bytes smaller than the existing untrained-MTP artifact ornith-ai/Ornith-1.5-35B-A3B-w8g32-q4g64-q5g64-q6g64-bf16.ninfer (22,783,246,080 bytes). The delta is the size difference between the model's own chat_template.jinja (embedded here) and the base pin's chat_template.jinja (embedded in the untrained-MTP artifact).

Benchmark

Measured on this host (RTX 5090, sm_120a), single-stream CLI, identical spec flags for both artifacts (--spec mtp --draft-tokens 5 --lm-head-draft --draft-adaptive --draft-adaptive-min 3 --prefill-chunk 4096 --vision), a 150-word prose prompt, --max-new 256, reproducible sampling (--temperature 0.6 --top-k 20 --top-p 0.95 --seed 0). The unfixed artifact is the native ornith-ai head; the fixed artifact is this card's trained shisa-ai head.

The fixed head drafts fewer tokens (351 vs 459) across fewer rounds (117 vs 154) but accepts more of them per round (2.17 vs 1.65 tok/round) — the signature of a stronger draft head that the target verifies more often, with less wasted speculative work.

Note. The shisa-ai head was trained with a code-heavy distribution, so the ~69% code / ~60% overall acceptance figures from the upstream repo are expected on code prompts. The numbers above are a prose prompt (the harder case for a code-tuned head) and therefore land lower; a code-generation prompt should approach the upstream figures. These prose numbers are the validated, reproducible baseline.

Serving

• Engine: NInfer — a custom C++ inference engine, built for CUDA 13, sm_120a.
• Served via: the NInfer engine.
• Model id: shisa-ai/Ornith-1.5-35B-A3B-MTP
• Capabilities: multimodal (text + image input); MTP (multi-token prediction) via the trained MTP head and the DFlash draft head.

Serving flags

--max-concurrency 2 \
--kv-capacity auto \
--kv-dtype int8 \
--spec mtp \
--draft-tokens 5 \
--lm-head-draft \
--draft-adaptive \
--draft-adaptive-min 3 \
--prefill-chunk 4096 \
--vision

Known Issues & Limitations

• No quality benchmark run. No KL-divergence or perplexity comparison has been run on this artifact. Quantization fidelity is not validated by a benchmark.
• Not numerically validated. No smoke/quality run against this artifact has been completed; treat quantization fidelity as unmeasured.
• Speculative-decoding output correctness on the trained head was not independently re-validated for this artifact.
• Template: N/A — the model's own chat_template.jinja is embedded (no mismatch).

License

The model base is released under the MIT license (the front-matter license: mit field follows the knoopx artifact-repo convention for the derived container). The grafted trained MTP head is a derivative of the Apache-2.0 Qwen/Qwen3.6-35B-A3B MTP head. The quantization/conversion tooling (the NInfer inference engine) governs any additional license terms on the engine itself.

Citation / Provenance

• Upstream repo: ornith-ai/Ornith-1.5-35B-A3B (BF16 source)
• Trained MTP head: shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY
• DFlash draft: z-lab/Qwen3.6-35B-A3B-DFlash
• Base front-end pin: Qwen/Qwen3.6-35B-A3B
• Inference engine: NInfer (CUDA 13, sm_120a)
• Converter: tools.convert.qwen3_6_35b_a3b.convert (recipe qwen3_6_35b_a3b-v2), groupwise-int4 mode