Ornith-1.5-35B-A3B-Vision-MTPLX-4bit

A validated oMLX 0.6.3 derivative of the 4-bit ornith-ai/Ornith-1.5-35B-A3B, combining the official vision tower with the native one-layer MTPLX head.

Ornith-1.5-35B-A3B is a 35B-parameter mixture-of-experts model activating approximately 3B parameters per token. This release preserves its native one-layer MTP head and includes the vision tower.

Quantization

• Main model: 4-bit affine, group size 64
• MTP runtime sidecar: 4-bit affine, group size 64 (mtp-omlx.safetensors)
• MTP norms: BF16 (non-quantized 1-D tensors)
• Vision runtime sidecar: BF16 (model-vision-omlx.safetensors)
• Architecture: Qwen3_5MoeForConditionalGeneration
• MTP contract: Qwen3-Next / delta norm encoding

The MTP sidecar contains 46 tensors, including 13 quantized matrix-weight tensors and 7 BF16 normalization tensors. All seven raw zero-centered RMSNorm gammas were converted to the MLX representation by adding 1.0. This conversion raised measured draft acceptance from 0% to 94-100% without changing deterministic output.

Validated oMLX settings

• Runtime: oMLX 0.6.3
• Native MTP: enabled
• Draft depth: 2 (recommended mixed vision/text default)
• Thinking: disabled for the controlled benchmark

Measured performance

Test environment

• Mac Studio with Apple M3 Ultra (28-core CPU) and 96 GB unified memory
• macOS 26.6.2
• oMLX 0.6.3
• Deterministic decoding with thinking disabled
• MTP depth 2 unless otherwise stated

All results below are local measurements. pp is the prompt-token count and tg is the requested generation-token count.

Single-request synthetic benchmark

Compared with MTP disabled, depth 2 increased single-request generation speed by 22.3% for pp1024 and 20.3% for pp4096. TPOT decreased by 18.1% and 16.8%, while end-to-end latency decreased by 12.7% and 6.7%, respectively. The tradeoff was approximately 1.70-1.75 GB more peak memory. TTFT and prefill throughput were effectively unchanged, with differences of about 1.2% or less.

Extended prompt-length sweep (single configuration — MTP Depth 2)

The rows below extend the single-request sweep to longer prompts at a fixed tg128. They use a single configuration (MTP Depth 2) and reproduce the Depth 2 baseline already shown above: the pp1024 / pp4096 single-config values (481.9 ms / 2,124.7 / 133.8 / 1.447 s / 21.05 GB, and 1,629.8 ms / 2,513.2 / 113.1 / 2.769 s / 23.01 GB) match the Depth 2 rows in the comparison table, to within measurement variation.

Continuous batching (pp1024/tg128)

The scale factor is relative to batch size 1 within the same run.

The multi-request rows are observational rather than a controlled measurement of MTP itself. During the run, oMLX reported that MTP was inactive for the two-row batch because standard batched decode was faster, and the enabled and disabled runs used different VLM/language-only load paths. Consequently, the batch-size 2 and 4 rows should not be interpreted as evidence that MTP causes a concurrency regression.

Continuous-batching sweep (single configuration — MTP Depth 2)

The sweep below re-runs a single batch-size series at the MTP Depth 2 configuration. The batch-size 1 row (129.0 → 133.8 tok/s generation, 480.2 → 481.9 ms TTFT, 1.480 → 1.447 s end-to-end) reproduces the Depth 2 baseline in the comparison table, to within measurement variation. The 2× and 4× rows carry the same caveat as the comparison table: they are a single configuration sweep, not a controlled Off-vs-Depth comparison, and the batched rows were run under a single load path.

Fixed-task warm depth sweep

The controlled text microbenchmark favored depth 3, but subsequent real-workload testing regressed after switching to depth 3. No exact result was retained for that later workload, so it is not quantified here. Depth 2 remains the final recommended default because it performed consistently across the measured text and visual tasks. These results are workload- and machine-specific, not a general performance guarantee.

Performance highlights (single configuration — MTP Depth 2)

The following observations describe the Depth 2 sweep above. They are local point measurements on this machine and oMLX setup, not a cross-hardware ranking.

• Throughput scales with prompt length. End-to-end aggregate throughput rises from ~796 tok/s at pp1024 to a peak of ~1,905 tok/s around pp16384, then settles at ~1,651 tok/s at pp32768. The fixed per-request generation cost (tg128) amortizes more effectively as the prompt grows, so long-prompt single requests are the strongest case.
• Memory stays bounded across prompt lengths. Peak memory climbs from ~21 GB at pp1024 to ~24 GB at pp32768 — roughly a 3 GB headroom over a 32 K token prompt — consistent with the 4-bit language body and the preserved BF16 vision tower.
• Fast first token. First-token latency stays sub-millisecond at short prompts (~482 ms TTFT at pp1024), and generation throughput holds in the 105–134 tok/s range across all prompt lengths.
• Continuous-batching headroom narrowed. The 4× batch now achieves ~2.03× aggregate generation throughput, lower than the ~2.13× recorded in the comparison table above. This is expected once per-request throughput is already high: there is less free throughput to recover by packing requests, so the batching speedup compresses. It is not a regression in per-request behavior.

Model intelligence benchmarks

The benchmarks above measure oMLX latency, throughput, and memory. This section measures model intelligence — accuracy on general-knowledge and coding benchmarks — which is a different axis, and was measured with Think reasoning enabled throughout.

Sampling windows: 100/14042 (MMLU), 100/11582 (CMMLU), 100/817 (TruthfulQA), 100/164 (HumanEval), 100/500 (MBPP), 100/1055 (LiveCodeBench). Recorded 2026-08-29.

Cross-model comparison (accuracy %)

Ornith-1.5-35B-A3B-Vision-MTPLX-4bit (Think on)

Highlights:

• Complex-code leader within the 1.5 series. The only 1.5-series model to exceed 60% on LiveCodeBench (61.0%), well ahead of the other 1.5 variants (abliterated 16.7%, MLX-4bit 23.3%, MLX-6bit 30.0%) and above Ornith-1.0 (53.3%). LiveCodeBench measures real-world, hard coding tasks, so the gap is meaningful.
• Strong code generation. HumanEval 93% and MBPP 91% — MBPP alone outperforms Qwen3.8-27B (81%).
• Balanced general benchmarks. MMLU 85% and TruthfulQA 89% sit close to Ornith-1.0 (Think on); Chinese knowledge (CMMLU 74%) is weaker than 1.0 (87%) but stronger than the other 1.5 variants (37–53%).
• Fair cross-comparison. vs Qwen3.8-27B-oQ4e-fp16-mtp (Think off): beats it on LiveCodeBench (61% vs 50%) and MBPP (91% vs 83%), ties MMLU (85%), slightly behind on HumanEval (93% vs 99%) and TruthfulQA (89% vs 90%). Net: stronger on hard real-time coding, comparable on general common sense.

Caveats:

• LiveCodeBench cost dominates: 4346.2s, roughly 10× MMLU — the Think overhead on complex-code reasoning is the main cost. Other benchmarks run at moderate latency.
• LiveCodeBench here uses a 100-sample window, larger than the 30-sample runs of the other 1.5 variants, so its 61.0% is statistically more reliable — but direct numeric comparison against those 30-sample figures should carry that caveat.
• CMMLU (74%) is meaningfully lower than Ornith-1.0 (87%); the 4-bit quantization trades some Chinese-knowledge fidelity.
• These are point measurements on this machine and oMLX build, not a general guarantee.

Usage

# Select this directory in oMLX 0.6.3, enable native MTP,
# and set the MTP draft-token count to 2.

The model can also be loaded by MLX-LM for autoregressive text generation, without MTPLX speculative decoding.

Provenance and licensing

• Base model: ornith-ai/Ornith-1.5-35B-A3B — MIT
• 4-bit MTPLX source: wang-yang/Ornith-1.5-35B-A3B-MTPLX-4bit
• Vision/config compatibility reference: scottlowry/Ornith-1.5-35B-A3B-oQ4e-mtp
• Runtime: MTPLX — Apache-2.0
• Conversion: MLX affine quantization, group size 64

This repository contains a quantized derivative of the base model. Please follow the upstream model license and usage conditions. MTPLX attribution is required when shipping a product or service built on MTPLX:

Powered by MTPLX
https://github.com/youssofal/MTPLX