Inkling-mlx-4bit (4-bit, omni / multimodal, self-contained MLX)
An MLX 4-bit build of Thinking Machines' Inkling (975B-total / 41B-active MoE),
quantized from the BF16 checkpoint for Apple Silicon. Omni: keeps the text decoder
plus the vision (HMLP) and audio (dMel) towers. Self-contained. This bundles the loader
package (inkling_mlx/), so it runs with just mlx + mlx-lm + transformers.
The higher-fidelity sibling of mlx-community/Inkling-mlx-2bit:
~548 GB (vs ~315 GB at 2-bit), trading size for sharper text and multimodal quality.
Quantization (recipe: experts_only)
- 4-bit affine group quantization (group size 64) on the routed experts and the vision / audio matmuls this is the bulk of the weights.
- Kept in bf16 (protected): attention, token / output embeddings, RMSNorms, the router gate, the per-layer short-convolutions, and the relative-position bias.
Usage
# pip install mlx mlx-lm transformers
# download this repo (it already includes the inkling_mlx/ loader)
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer
model, config = load("./Inkling-mlx-4bit")
tok = AutoTokenizer.from_pretrained("./Inkling-mlx-4bit", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
Running on a single Mac with SSD expert-offload
At ~548 GB this doesn't fit resident on one Mac but an MoE only fires 6 of 256 experts per token. So keep the always-needed weights in RAM (attention, shared experts, embeddings, norms, router, vision / audio towers) and page the routed experts from SSD on demand, letting the OS page cache hold the hot ones. This runs the omni build on a single Mac Studio.
Sample text decoding (4-bit, greedy)
| Prompt | Generated continuation | Notes |
|---|---|---|
The capital of France is | Paris. The capital of Italy is Rome. The capital of Spain is Madrid. The capital of Russia is Moscow. | ~0.33 tok/s, SSD expert-offload |
Tool: github.com/huckiyang/mlx-moe-offload
(MIT) — a drop-in over mlx-lm's SwitchGLU.
pip install mlx mlx-lm scipy pillow
git clone https://github.com/huckiyang/mlx-moe-offload && cd mlx-moe-offload && pip install -e .
# 1) one-time: repack the stacked experts into a per-expert SSD store
python -m mlx_moe_offload.repack --build /path/Inkling-mlx-4bit --out /path/Inkling-mlx-4bit-offload
OFF=/path/Inkling-mlx-4bit-offload
# 2) generate — text, image, or audio (omni)
python examples/inkling_omni.py --offload-dir $OFF --prompt "The capital of France is"
python examples/inkling_omni.py --offload-dir $OFF --image cat.png --prompt "What is in this image?"
python examples/inkling_omni.py --offload-dir $OFF --audio q.wav --prompt "Transcribe in English:"
Decode speed is gated by the routing cache hit-rate and SSD bandwidth, please check store.stats().
Attribution
- mlx-moe-offload from huckiyang/mlx-moe-offload
- Weights converted from
thinkingmachines/Inkling. - The bundled
inkling_mlx/loader is vendored from PipeNetwork/inkling-mlx (Apache-2.0, Copyright 2026 PipeNetwork) and David.
LICENSE + THIRD_PARTY_NOTICES.md are included in this repo.