Ornith-1.5-35B-A3B — REAP-192 · mxfp4 · MLX (tuned for 24 GB Apple Silicon)
A compressed derivative of ornith-ai/Ornith-1.5-35B-A3B built to run fast, swap-free, single-user inference on a base M4 Mac mini with 24 GB of unified memory — a machine on which the plain 4-bit conversion (19.5 GB) does not run at all (hard Metal out-of-memory against the ~18 GB working-set cap).
This repo is the recommended quality-preserving build: 49.8 tok/s decode with task quality measurably above an uncompressed-expert 3-bit reference. A max-speed variant reaching 51.2 tok/s (at lower quality) is documented below and reproducible with the included scripts.
| Decode | Prefill (8k ctx) | Warm TTFT | Task quality* | NLL | Weights | |
|---|---|---|---|---|---|---|
| This build (recommended) | 49.8 tok/s | 451 tok/s | 0.31 s | 0.9814 | 0.8950 | 14.2 GB |
| Max-speed variant (recipe below) | 51.2 tok/s | 448 tok/s | 0.31 s | 0.9314 | 0.9039 | 14.1 GB |
| Reference anchor (unpruned, 3-bit experts) | 36.7 tok/s | 410 tok/s | 0.43 s | 0.9543 | 0.8977 | 16.1 GB |
*Composite pass-rate over a 42-task executable suite (unit-tested coding, exact-answer math, JSON tool calls, schema-validated structured output, format compliance, stability checks) plus rotated held-out task generations; NLL measured on a fixed mixed code/prose corpus. All numbers measured on a base M4 Mac mini (24 GB, macOS 26.2) serving over an OpenAI-compatible HTTP endpoint, median of repeated runs, greedy decoding. An independent fresh-context audit reproduced the recommended line bit-exactly on quality/NLL and within 0.2 % on decode.
What exactly was done (lineage)
Source: ornith-ai/Ornith-1.5-35B-A3B-MLX @ revision 1713baf2e38984a3c557195caf167759144d1d7e
(bf16, 69.3 GB). The Ornith team's card notes Ornith-1.5 was developed on top of Qwen3.5
and Gemma4 lineages; see the original model card
for training details and benchmark results of the uncompressed model.
Transformations, in order (each reproducible with the scripts in this repo):
- REAP expert pruning, 256 → 192 experts per layer (25 %), following REAP: Router-weighted Expert Activation Pruning (Lasby et al., Cerebras, 2025): saliency = mean over routed tokens of (router gate probability × ‖expert output‖₂); the 64 lowest-saliency experts per layer are removed and the router renormalizes over survivors. Saliency was collected on-device over ~295k mixed-domain calibration tokens (code, prose, math, chat, tool-call JSON). The ranking is robust: recollection on an independently seeded calibration mix changed the survivor set by <1 % (top-192 Jaccard 0.992). Shared experts are never pruned.
- Quantization (mxfp4 mode): routed experts, attention, and embeddings in mxfp4 (group 32); shared expert in 8-bit affine (it runs on every token — this is what lifts quality above cheaper builds); router gates in 8-bit affine. Effective ~4.33 bits/weight.
- Top-k reduced 8 → 7 (
num_experts_per_tok: 7inconfig.json): −12.5 % active bytes per token for +0.55 % NLL. k=6 was measured and rejected (three variants, all fail quality gates).
No fine-tuning, no weight editing beyond the above; the tokenizer and chat template are unchanged from the source.
Use it
Requires an Apple-silicon Mac (≈16 GB free memory; built for 24 GB machines) and mlx-lm ≥ 0.31:
pip install -U mlx-lm
# generate
mlx_lm.generate --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX \
--prompt "Write a Python function that merges overlapping intervals." --max-tokens 400
# OpenAI-compatible server (stock)
mlx_lm.server --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX --port 8080
Optional +8–10 % serving throughput: the stock server spends ~2.4 ms/token in its
batching/parsing layer. scripts/minimal_server.py is a drop-in single-user replacement
(same OpenAI-compatible API, adaptive prompt-prefix caching, reasoning/content split)
that serves this model at the bare-API decode rate:
python scripts/build_minimal_server.py # builds a patched runtime copy next to your mlx-lm
PYTHONPATH=./patched-minsrv python -m mlx_lm server --model <this-model> --port 8080
Max-speed variant (51.2 tok/s)
Quantize everything (including the shared expert) in mxfp4 instead of step 2 above —
recipe mxfp4 in scripts/quantize_variant.py, then the same pruning and k=7 config.
Costs ~5 task-quality points (0.9814 → 0.9314) for +1.4 tok/s; measured details in the
table. We recommend this build instead unless raw decode is all that matters. (A middle
build, 8-bit shared + 3-bit head, lands at 50.8 tok/s / 0.9564 — recipe mxfp4-sh8h3.)
Measured behavior on the target machine
- Peak memory ≈ 14.3 GB; zero sustained swap; macOS memory pressure stays normal; no thermal derating over sustained generation (decode-rate decay 0.99 over 1024 tokens).
- Prompt-prefix reuse (e.g. stable system prompts) cuts TTFT ~25× under the included server; cold start (process spawn → first token) ≈ 8 s.
- Context: KV cache is tiny for this hybrid architecture (10 of 40 layers are full attention; ~20.5 kB/token) — long contexts are memory-cheap; 8k-token prompts prefill at ~450 tok/s.
Limitations
- Compression is not free. The recommended build's task-suite composite (0.9814) and NLL (0.8950) sit at-or-above the measured uncompressed-expert reference on this suite, but the suite is finite (coding/math/tool/structured/format/stability, English); expect some degradation vs the bf16 original on knowledge-heavy or multiple-choice tasks — REAP's authors measured exactly that pattern at 25 % pruning. The bf16 original could not be run on any available hardware for direct comparison.
- k=7 routing and expert pruning shift the output distribution; borderline greedy tokens can differ from the source model's.
- Tuned for single-user serving; the included minimal server is deliberately not a multi-user production server.
- Measured on macOS 26.2 / mlx 0.32.2 / mlx-lm 0.31.3; other versions may differ.
Safety
This derivative changes efficiency characteristics only: no fine-tuning was performed and no safety-relevant components were intentionally altered. It inherits the behavior, alignment properties, and content policies (and their gaps) of Ornith-1.5-35B-A3B — refer to the original card. Compression can subtly shift refusal boundaries and factual reliability; the quality gate used here measured task competence, not safety behavior. Apply your own safeguards for user-facing deployments, and do not treat outputs as authoritative without verification.
Reproduce / verify
scripts/ contains the exact tooling: quantize_variant.py (named recipes, pinned source
revision), reap_collect.py + reap_prune.py (saliency + pruning; prune-of-quantized is
exact because per-expert quantization groups are independent), build_minimal_server.py /
minimal_server.py (serving runtime), and DERIVATION.json (machine-readable provenance
for these exact weights). Deriving from scratch needs the 69.3 GB source checkpoint and
any Apple-silicon Mac with ~90 GB free disk (conversion is shard-streamed; 16 GB RAM
suffices).
Attribution & citations
- Base model: Ornith-1.5-35B-A3B by the Ornith Team (MIT) — please cite their release:
@misc{ornith_1_5, title={{Ornith-1.5}: From Self-Scaffolding to Self-Improvement}, url={https://ornith.ai/ornith_1_5.html}, author={{Ornith Team}}, year={2026}} - Pruning method: REAP the Experts (Lasby et al., 2025); independent reimplementation in
scripts/. - Runtime: MLX and mlx-lm (Apple, MIT).
- Compression, evaluation, and serving work by mchen04, 2026.
License: MIT, inherited from the base model's declared license; see LICENSE.