Ternary Bonsai 27B — OpenVINO INT4 SYM
This is an OpenVINO INT4 conversion of PrismML's Ternary-Bonsai-27B unpacked checkpoint, packaged for Intel CPU/GPU inference.
PrismML created and trained Ternary Bonsai. This repository contains only the OpenVINO conversion by Wondernutts. See the original model card for the authoritative description and intended use.
Important: this is not the native ternary build
The source checkpoint expands the ternary model into ordinary FP16 tensors for compatibility. This repository applies conventional OpenVINO INT4 compression to those unpacked tensors; it does not use PrismML's native ternary packing or custom kernels.
For the efficient native deployment, use PrismML's ternary MLX or ternary GGUF release.
Conversion details
| Item | Value |
|---|---|
| Source | prism-ml/Ternary-Bonsai-27B-unpacked |
| Architecture | Qwen3.5 conditional-generation/VLM graph |
| Weight compression | INT4 symmetric, group size 128 |
| Backup precision | INT8 symmetric |
| INT4 ratio | 1.0; all language-model layers requested |
| OpenVINO binary payload | approximately 13.94 GiB |
| Export stack | Optimum 2.2 development exporter, Transformers 5.2, OpenVINO IR |
The repository includes language, text-embedding, vision, tokenizer, and detokenizer IRs. The source configuration's maximum context has not been independently validated for this conversion.
Download
Download the public repository:
python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/Ternary-Bonsai-27B-int4-sym-ov --local-dir ./ternary-bonsai-27b-int4-ov
Run on Intel GPU
import openvino_genai as ov_genai
model_dir = "./ternary-bonsai-27b-int4-ov"
pipe = ov_genai.VLMPipeline(model_dir, "GPU", CACHE_DIR="./ov_cache")
config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.8
config.top_p = 0.95
pipe.start_chat("You are a thoughtful and precise assistant.")
print(pipe.generate("Write a short scene set in a city floating above the clouds.", generation_config=config))
pipe.finish_chat()
The shipped tokenizer applies the model's chat template. Change "GPU" to "CPU" for CPU inference. For continuous batching on Intel GPU, set DYNAMIC_QUANTIZATION_GROUP_SIZE=0 if dynamic activation quantization causes garbled output.
For image input, load an RGB image into an openvino.Tensor and pass it as images=[tensor] to VLMPipeline.generate(). Vision and long-context behavior have not been independently benchmarked for this build.
License and attribution
Apache-2.0, following the source repository. Model authorship and training credit belong to PrismML; OpenVINO conversion credit belongs to Wondernutts.