inkling-tiny-preview

[!WARNING] Research preview — not extensively tested. This model has not been extensively evaluated for coherence or alignment/safety. It can and will produce text that is incoherent, factually wrong, nonsensical, or otherwise unsafe. It is not suitable for any production or user-facing use. It is shared for research and curiosity only. Do not rely on its outputs. Unofficial and not affiliated with or endorsed by Thinking Machines.

A heavily-pruned, LoRA-healed, 4-bit MLX derivative of thinkingmachines/Inkling-Small, built to run on a 16 GB Apple-silicon Mac.

What it is

Inkling-Small is a large multimodal Mixture-of-Experts model (256 routed experts per MoE layer). This preview keeps only 8 of those 256 experts (a ~97% cut of the routed parameters), quantizes the result to 4-bit in MLX, and then LoRA-heals the wreckage on a small general-English corpus so it produces coherent text again. It is text-only (the vision/audio towers are dropped).

The result is a small (~8 GB) model that generates fluent, structured English — but with tiny capacity and no reliable world knowledge. It will confidently state false things (e.g. that the capital of France is "1.5 million") and cannot do arithmetic. Think of it as a demonstration that an aggressively-pruned MoE can be revived into coherence on consumer hardware — not as a useful assistant.

How it was made

1. Prune — stream only 8 experts/layer from the original checkpoint (top-6-of-8 routing kept).
2. Quantize — 4-bit affine (group size 64) in MLX.
3. Heal — LoRA (rank 16) on attention + expert projections, trained on ~6M tokens of FineWeb-Edu. Best checkpoint at cross-entropy ≈ 2.23 (perplexity ≈ 9). The adapter is shipped separately from the base because fusing it into the 4-bit weights measurably degrades quality.

Usage

Apple-silicon Mac required (MLX).

pip install mlx mlx-lm transformers numpy
python run.py --prompt "The three primary colors are"

Greedy decoding loops badly on a model this small; run.py applies a repetition penalty (default 1.3), which you should keep. Layout: 4-bit base weights + adapter/adapters.safetensors (LoRA) + bundled inkling_mlx/ runtime + run.py.

Example output (--prompt "The three primary colors are"):

The three primary colors are the red, white and black. The other two include yellow.
These four shades of color have been used in a variety of applications:
- Red: Used to indicate danger or urgency
- White: ...

Fluent and structured — and also wrong (the primary colors are not "red, white and black"). That is representative: coherent in form, unreliable in fact.

Limitations

• Not safety- or alignment-tested. No red-teaming, no refusal training, no evaluation. May emit false, biased, offensive, or otherwise harmful content.
• Not factual. ~97% of the routed experts were deleted; world knowledge is largely gone.
• Narrow. Healed on a small general-web sample; tends toward generic prose.
• English, text-only.

Attribution & license

• Base model: thinkingmachines/Inkling-Small (Apache-2.0).
• Bundled inkling_mlx/ runtime: from mlx-community/Inkling-Small-mlx (Apache-2.0), with a one-line modification in moe.py (stop_gradient on expert indices) so the model can be LoRA-trained; identity at inference.

Released under Apache-2.0, inheriting the base model's license. This is a modified derivative work; it is unofficial and not affiliated with, sponsored by, or endorsed by Thinking Machines.