Ternary-Bonsai-1.7B — LiteRT-LM (INT2 / ternary, GPU + CPU)
Ternary (INT2) .litertlm bundles of
prism-ml/Ternary-Bonsai-1.7B-unpacked,
packaged for the LiteRT-LM runtime and
Google AI Edge Gallery.
These run on the GPU backend (recommended — fastest) or the CPU backend. Not the NPU.
Recommended:
bonsai-1.7b-int2pc-4k-gpu.litertlm. The plain, unmodified build with no optimizations applied. Everything else in this repo is experimental — longer context or lighter activations, but whether a given one loads depends on your app's LiteRT / LiteRT-LM version and dependencies. Start with the recommended build; reach for an experimental one only if you specifically need what it offers.
Builds
| file | ctx | weights | activations | notes |
|---|---|---|---|---|
bonsai-1.7b-int2pc-4k-gpu.litertlm | 4096 | INT2 per-channel | fp32 | RECOMMENDED — plain, no optimizations |
bonsai-1.7b-v2max-sdpa-fp16-int2all-32k-fixed2.litertlm | 32768 | INT2 (all) | fp16 | experimental — fp16 activations, fused SDPA |
bonsai-1.7b-v2sdpa-int2all-32k-fixed2.litertlm | 32768 | INT2 (all) | fp32 | experimental — fused SDPA |
bonsai-1.7b-int2pc-32k-crashfix.litertlm | 32768 | INT2 per-channel | fp32 | experimental — long context, no SDPA fusion |
bonsai-1.7b-int2pc-32k-mp-crashfix.litertlm | 32768 | INT2 per-channel | fp32 | experimental — multi-prefill buckets |
32768 is the model's full context — max_position_embeddings for Ternary-Bonsai-1.7B is 32768,
so the long-context builds here sit at the ceiling rather than truncating it.
All bundles carry Bonsai's own chat template (Qwen3 ChatML with the reasoning block intact).
Why several variants
The GPU accelerator runs a float graph — INT2 is a storage format, and compute happens in fp16/fp32. Two consequences shape this list:
- Per-channel ternary dequantizes coherently. Block-quantized weights mix scales inside a single GEMM, which is why the per-channel builds are the conservative choice.
- fp16 activations are lighter but less widely supported. The
fp16bundles are the most likely to meet a runtime that won't take them.
Support ranges by app, so the full set is published rather than a single "best" build.
Backends
| backend | status |
|---|---|
| GPU | ✅ recommended, fastest |
| CPU | ✅ works — verified on a Pixel 9 Pro XL's CPU backend (recommended build) and on a desktop x86 CPU via the litert-lm Python package |
| NPU | ❌ not supported by these bundles |
Why CPU works even though XNNPACK has no INT2 kernel: XNNPACK takes the float ops, and the INT2
FULLY_CONNECTED ops fall back to LiteRT's built-in kernels, which handle INT2 directly. Expect CPU to be
noticeably slower than GPU. A YNNPACK-enabled runtime could speed up CPU INT2, but it isn't required.
Sampling defaults
Every bundle ships these in its LlmMetadata, so a LiteRT-LM host picks them up without
any configuration:
| parameter | value |
|---|---|
| type | TOP_P |
| top-k | 20 |
| top-p | 0.85 |
| temperature | 0.5 |
These are the values the bundles were built with. Override them in your host if you want different behaviour.
Usage
Any LiteRT-LM host — the AI Edge Gallery app, or litert_lm_main — with the GPU backend (fastest) or the CPU backend selected.
Provenance and licensing
- Base model:
prism-ml/Ternary-Bonsai-1.7B-unpacked, Apache-2.0 - Built from Qwen3-1.7B, Copyright 2024 Alibaba Cloud, Apache-2.0
- These bundles are quantized, repackaged derivatives. No weights were retrained.
Created using Bonsai by Prism ML.
Training data: None was used here. These are post-training quantizations and repackagings of the released Bonsai checkpoint; no additional training, fine-tuning, or calibration data was involved. For the base model's training data, see the upstream Prism ML and Qwen3 model cards.
PII: No dataset was collected, processed, or shipped as part of this conversion, so no personally identifiable information is present in these artifacts beyond whatever the upstream released weights already encode.