Bonsai 2 Ternary 9B: MLX
A 9B-class vision-language model with ternary transformer weights, packed for MLX on Apple Silicon.
3.46 GB on disk, language model + vision tower (FP16 language model: ~17.9 GB)
Highlights
- Ternary language weights stored in MLX's 2-bit grouped format (group 128). MLX stores a scale and a bias per group, so the container costs a little more than the GGUF packings; the weights decode to exactly the same ternary values.
- Vision input through the official vision tower, carried unrotated in FP16.
- 262K-token context, inherited from the Qwen3.5-9B hybrid-attention backbone (24 linear-attention and 8 full-attention layers).
Model Overview
| Item | Specification |
|---|---|
| Base model | Derived from Qwen3.5-9B (architecture unchanged) |
| Parameters | ~8.95B language model (32 blocks, hidden 4096) + ~0.46B vision tower |
| Weight format | Ternary {−1, 0, +1} in MLX affine 2-bit, group 128 |
| Weight basis | Blockwise Hadamard rotation folded into the stored weights; the bundled runtime applies the matching transform to activations |
| Backends | MLX on Apple Silicon (Metal): mlx-vlm 0.7.8+ natively, or the loader in runtime/ |
| License | Apache 2.0 |
Files
| File | Contents | Size |
|---|---|---|
model.safetensors | language model (2-bit, group 128) + vision tower (FP16) | 3.46 GB |
mtp/ | multi-token-prediction draft head, 4-bit (optional, see below) | 0.14 GB |
runtime/ | a bundled loader for older mlx-vlm versions, plus pinned requirements.txt | |
quickstart.py | one-command test with the recommended sampler | |
PACK-RUNTIME.md | the bundled loader's contract and limits |
Quickstart
PrismML-Eng/Bonsai-demo is the source of truth for running these models. Where anything here disagrees with it, it is right.
hf download prism-ml/Ternary-Bonsai-2-9B-mlx-2bit --local-dir bonsai2-9b-mlx
pip install "mlx-vlm>=0.7.8"
mlx-vlm 0.7.8 and newer load this pack natively (model_type: prism_hadamard_qwen35):
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("bonsai2-9b-mlx")
prompt = apply_chat_template(processor, model.config, "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=4096,
temperature=1.0, top_p=0.95, top_k=20))
For an OpenAI-compatible server, run python -m mlx_vlm.server --model bonsai2-9b-mlx and send the sampling settings in each request.
The released mlx-lm (0.32.0) cannot load this pack yet; its support is merged and arrives with the next mlx-lm release. On older mlx-vlm versions, use the loader bundled in runtime/ (pip install -r bonsai2-9b-mlx/runtime/requirements.txt, then python bonsai2-9b-mlx/quickstart.py "your prompt"); PACK-RUNTIME.md documents it. quickstart.py is a smoke test: it answers with thinking off and stops at 256 tokens.
Pass no images for text-only use.
For CUDA, CPU, and llama.cpp on Metal, use the GGUF packs of the same weights: Ternary-Bonsai-2-9B-gguf.
Best Practices
Generation Parameters
We recommend these sampling settings (the same as the GGUF packs):
- Thinking mode (default):
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=1.0,repetition_penalty=1.0- Non-thinking mode:
temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0
Pass the sampler explicitly. mlx-vlm's Python generate() decodes greedily unless you pass these arguments. mlx_vlm.server takes temperature, top_p and top_k from generation_config.json when a request omits them, but not presence penalty or min_p, so send the full set per request. Apply penalties over the last 64 tokens where your client supports a penalty window. Avoid greedy decoding with thinking on; it can get stuck reasoning on some prompts. To turn thinking off, render the chat template with enable_thinking=False.
Output Length
Give thinking mode room to finish: even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off.
System Prompt
You can use a simple system prompt such as:
You are a helpful assistant
Speculative Decoding (optional)
mtp/ holds this model's trained multi-token-prediction head as an mlx-vlm drafter. The main model verifies every drafted token, so the output matches plain decoding (up to float16 near-ties). It helps most on copy-heavy work such as code edits and rewrites; on general chat it is about as fast as plain decoding or slightly slower, so plain decoding stays the default. MTP.md has the code. It was validated with the bundled runtime on mlx-vlm 0.6.3 and has not yet been re-checked on 0.7.8.
License
Apache 2.0, following the base model.