MiniCPM5-2B — JANG_8M

8-bit MLX quantization of openbmb/MiniCPM5-2B, built with the full calibrated pipeline: imatrix + AWQ + GPTQ on 1.85M calibration tokens.

2.49 GiB · 8.501 bits/weight · median KL 0.0055 bits vs bf16 · 97.25% top-1 agreement

Loads with stock mlx_lm — no custom model code, no trust_remote_code.

JANG_8M vs stock MLX 8-bit

Both quantized from the same bf16 source at the same width (affine 8-bit, group_size 64), measured on the same held-out prompts on an M5 Max.

At identical size and identical speed, JANG_8M is 1.86× closer to the source distribution. That gap is what the calibration buys — it is not a size-versus-quality trade.

Speed is a tie with stock MLX, as it should be: same width, same group size, same kernels. The 57% decode gain over bf16 is the quantization itself.

Alignment

Stock mlx_lm.convert leaves 296 of 973 tensor payloads at offsets that do not match their dtype's natural alignment. MLX/Metal cannot expose those as zero-copy typed buffers and allocates a resident aligned copy of each at load. Every shard here is written aligned — 0 misaligned — verified as a hard gate before publishing.

How it was built

One calibration sweep over 1,849,995 tokens (1.68M prompt + 173K of the model's own greedy generations, so its thinking-mode distribution is in-calibration) drawn from a coding-weighted domain mix — coding 35%, agentic 20%, academic 15%, general 12%, Chinese 10%, long-context 4%, science 2%, security 2% — rendered through the model's real chat template across the thinking, non-thinking and tool-framed presets.

That one pass produces all three calibration signals:

• AWQ (α=0.25) folds into input_layernorm → q/k/v and post_attention_layernorm → gate/up — 84/84 fold sites, each proven function-preserving and proven to have actually moved the norm. o_proj and down_proj are deliberately not folded: o_proj's input channels are 4× tiles of v-rows under 16q/2kv GQA, and down_proj's input is the SwiGLU product whose gate side is nonlinear.
• GPTQ error-compensated rounding onto the fixed grid, float64 factorization with escalating damping, per-tensor best-of-RTN guard. Shipped on 295/296 tensors; embed_tokens stays RTN by design (a lookup table has no input Hessian). The down_proj Hessian is overdetermined 301× over its 6144 input dimension.
• bfloat16 scales. MLX computes affine quantized_matmul at promote(x.dtype, result_type(scales, biases)). This is a bf16 model, so float16 scales would promote every matmul to float32 and disqualify the fused bf16 kernels downstream. Verified: logits come back bfloat16.

Margin-conditioned flip curve

Where the quantization disagrees with bf16, conditioned on how decided the source model was (top1 − top2 logit margin):

A 300× drop from the undecided bin to the confident ones: the disagreements land where the source model was already a coin flip, not where it was sure. (The last two bins are 2 flips each — the 0.05→0.09% step is counting noise, not a rise.)

Usage

from mlx_lm import load, generate

model, tokenizer = load("OsaurusAI/MiniCPM5-2B-JANG_8M")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 84 * 3 / 2?"}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))

Sampling

The source model card's recommendation, declared in both generation_config.json and jang_config.json:

temperature = 1.0
top_p       = 0.95

Reasoning

The chat template takes enable_thinking:

prompt = tokenizer.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=True)

Tool calling — read this

MiniCPM5 emits XML tool calls, not JSON:

<function name="get_weather"><param name="city">Seoul</param></function>

Values containing <, & or newlines are wrapped in <![CDATA[...]]>.

⚠️ mlx_lm cannot infer a parser for this dialect. Its _infer_tool_parser matches this template against none of its built-in literals, returns None, and every tool call is passed through as raw assistant text with no error. A parser is required, and one ships with this bundle's tooling:

from jang_tools.minicpm5.tool_parser import install, parse_tool_calls
install()                       # must run BEFORE mlx_lm.load()
model, tokenizer = load("OsaurusAI/MiniCPM5-2B-JANG_8M")

Or parse directly:

parse_tool_calls(output, tools)
# [{'name': 'get_weather', 'arguments': {'city': 'Seoul'}}]

jang_config.json → tool_calling carries the dialect, the delimiters and the parser name so a non-Python runtime can wire its own.

Verified before publishing

Model

Quantized by Jinho Jang — eric@osaurus.ai · osaurus.ai

Base model © OpenBMB, Apache-2.0.