Inkling-Small-mlx-3bit
An MLX 3bit build of thinkingmachines/Inkling-Small
— 264 B total parameters, ~12 B active; 42 layers, 256 routed experts (top-6) + 2 shared;
natively multimodal (image + audio in, text out).
Sized to run fully resident on a 128 GB Mac — no expert-offload, no expert pruning, no layer streaming. The whole model sits in unified memory and decodes at conversational speed.
Why this model is the interesting one for Apple Silicon: speed follows the active parameter count, not how hard you compress. Inkling-Small activates ~12 B params per token, so this tier reads roughly 9.8 GB per token. The 975 B Inkling, by contrast, needs SSD expert-offload even at 2-bit and manages ~0.2–0.4 tok/s.
Tiers
Pick the tier that fits your machine. Sizes are on-disk and measured from the built artifacts; peak/speed columns are filled in only where a benchmark has actually run.
| tier | target Mac | recipe | on-disk | peak | load | prefill tok/s |
|---|---|---|---|---|---|---|
| 4bit | 192 GB | experts 4-bit, non-expert 8-bit | 153.5 GB | — | — | — |
| 3bit ← this repo | 128 GB | experts 3-bit, non-expert 8-bit | 120.9 GB | — | — | — |
| 2bit | 96 GB | experts 2-bit, non-expert 8-bit | 88.4 GB | — | — | — |
Speeds are not yet measured. On-disk sizes above are real (taken from the built artifacts), but load/peak/tok-s columns stay empty until
serving/bench_mlx.pyhas run on the target hardware — we would rather ship a blank column than a guess. Run it yourself with the command below and please open a discussion with your numbers.
Why MLX on Apple Silicon
MLX is Apple's array framework for Apple Silicon. For a big sparse MoE like this one, the practical differences from a CPU/GPU-split runtime are:
| what it means here | |
|---|---|
| Unified memory | The GPU reads the same DRAM as the CPU, so a 120 GB build needs 120 GB of system memory — not VRAM plus a host copy. There is no PCIe transfer per layer, which is what makes a >100 GB model practical on a desktop at all. |
| Lazy, mmap'd loading | mx.load memory-maps safetensors, so weights page in on demand instead of being read and copied up front. |
| Native quantized matmul | Quantized weights are multiplied in their packed form (quantized_matmul, and gather_qmm for MoE expert gathers) rather than dequantized to fp16 first — so low-bit tiers save bandwidth at runtime, not just on disk. |
| Per-module precision | nn.quantize(..., class_predicate=...) lets one checkpoint mix bit-widths per tensor class, which is exactly how these tiers keep attention and the router high-precision while the experts go low. |
| Small dependency surface | pip install mlx mlx-lm and a single Python model file. No compile step, no separate server binary. |
Honest limits: MLX's ecosystem is younger than llama.cpp's, it has no importance-matrix
("i-quant") style calibrated quantization yet, and it runs on Apple Silicon only. If you want a
GGUF build instead, one exists at
unsloth/Inkling-Small-GGUF.
Memory notes
peak (once measured) is peak unified-memory use, not file size — leave headroom for the KV
cache and the OS. Two things matter on a machine near its limit:
- Raise the Metal wired limit. It defaults to ~75% of RAM, which is below this tier's
footprint on a 128 GB machine:
sudo sysctl iogpu.wired_limit_mb=180000 # ~180 GB on a 192 GB Mac; scale to your RAM - macOS will kill a process that stays too large for too long (jetsam), even while memory pressure looks fine. If long generations die silently, move down a tier.
Quantization recipe
experts_only: the routed experts (and the vision/audio matmuls) carry the tier's bit-width,
while attention, token/output embeddings, RMSNorms, the router gate, the per-layer
short-convolutions and the relative-position bias stay high-precision (8-bit).
That asymmetry is the point, and it is nearly free: non-expert weights are only 5.9 B of the 264 B params (~6 GB at 8-bit), yet they are read in full on every token, while just 6 of 256 experts are. Bits spent there cost ~2% of the file and protect the paths every token depends on. Expert precision degrades gracefully; attention and router precision degrade globally.
Usage
# pip install mlx mlx-lm transformers (+ scipy for audio, pillow for images)
# this repo bundles the inkling_mlx/ loader, so there is nothing else to install
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer
path = "./Inkling-Small-mlx-3bit"
model, config = load(path)
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
Images are cut into 40 px patches (one soft-token each) and audio is d-mel encoded at 20
tokens/s; both go through InklingProcessor. See the source repo for a multimodal runner.
Benchmark it yourself
python serving/bench_mlx.py --model ./Inkling-Small-mlx-3bit --tier 3bit
Reports load time, prefill tok/s, decode tok/s at several context lengths, and peak memory.
Attribution
- Base model:
thinkingmachines/Inkling-Small, Apache-2.0. - The bundled
inkling_mlx/loader is vendored from PipeNetwork/inkling-mlx, Apache-2.0. - Quantized with
mlx; see the recipe above for exactly which tensors were touched.