Nanbeige 4.2 3B — CMF q4t (2.4 GB, one file, no Python)
Nanbeige/Nanbeige4.2-3B converted to the CMF format: a single mmap-served file executed by a pure-Rust runtime.
Nanbeige 4.2 is a Looped Transformer: 22 physical layers applied twice (44 virtual layers) with a re-norm at each loop boundary — the engine executes the loop natively — and a reasoning model that thinks before it answers.
Speed
Measured on a fanless MacBook Air (Apple M4) — 10-core CPU (4 performance + 6 efficiency), 10-core GPU, 24 GB unified memory, macOS 27. No external cooling, no overclock, nothing plugged in.
| GPU (Metal) | CPU only | |
|---|---|---|
| Decode, 64-token context | 22.4 tok/s | 9.2 tok/s |
| Decode on a discrete card (RTX PRO 6000, Vulkan, cortiq 0.5.49) | 94.2 tok/s | 23.1 tok/s (48-core) |
| Decode, 512-token context | 17.6 tok/s | — |
| Decode, 1024-token context | 14.2 tok/s | — |
| Prefill (prompt ingest) | 181 tok/s @ 512, 143 @ 1024 | 86 tok/s |
| Time to first token, 512-token prompt | 2.7 s | 6.1 s |
| RAM while running | ~2.6 GB + KV |
Method: cortiq bench --core (greedy, no repetition penalty — the
llama-bench contract), median of three runs on an idle machine. The GPU
column is native Metal, enabled with CMF_GPU=1. Needs cortiq
≥ 0.5.35 — earlier releases decode this file at ~9 tok/s and take 40 s
to ingest a 512-token prompt.
Why a "3B" runs at these rates and not faster. The loop means every token pays for 44 layer passes, ~6.3 G multiply-accumulates — the arithmetic of a 6B dense model. The 166k vocabulary with untied embeddings adds another 1.0B parameters, so the file really holds 4.17B. At q4t that is ~3.9 GB of weights read per token, and an M4 moves about 120 GB/s, which puts a hard ceiling near 31 tok/s on this machine no matter how good the kernels are. 22.4 tok/s is ~72% of that ceiling. A laptop with more memory bandwidth (M4 Pro/Max, or a discrete GPU through the Vulkan backend) scales roughly with it.
Run it
cargo install cortiq-cli # pure Rust, no Python; needs >= 0.5.35
hf download infosave/Nanbeige4.2-3Bcmf nanbeige42-3b-q4t.cmf --local-dir .
CMF_GPU=1 cortiq run nanbeige42-3b-q4t.cmf \
--prompt "What is the capital of France?" --max-tokens 300
CMF_GPU=1 cortiq serve nanbeige42-3b-q4t.cmf # OpenAI-compatible API
CMF_GPU=1 selects native Metal on Apple Silicon; on Linux/Windows the
same flag picks the Vulkan/DX12 backend. Without it the runtime stays on
the CPU, which is the second column above.
Reasoning models deliberate first — give them a few hundred
--max-tokens to reach the final answer.
Reproduce the conversion
cortiq convert --model Nanbeige/Nanbeige4.2-3B --quant q4t --output nanbeige42-3b-q4t.cmf
The converter streams the checkpoint shard by shard, so the free-disk requirement is one shard plus the output.
Fidelity
The looped forward is checked against a float32 PyTorch reference built from the upstream modeling code: same argmax and same top-5 ordering on the same prompt, with the q4t logits within ~2 of the reference's. The model's own quirks survive the quantization intact — including a reasoning opener it is very sure about.
License
Apache-2.0, inherited from the base model. Weights © Nanbeige LLM Lab; this repackaging only changes the storage format.
Run it on a phone
CMF Mobile is a Flutter
app built on this runtime: install it, point it at this 2.4 GB .cmf
and chat on-device with streaming tokens — or flip on its server mode
and the phone becomes an OpenAI-compatible endpoint for your network.
The engine, converter and format spec live at
https://github.com/infosave2007/cmf.