Binary Bonsai 27B — OpenVINO INT4 SYM
This is an OpenVINO INT4 conversion of PrismML's Bonsai-27B unpacked checkpoint, packaged for Intel CPU/GPU inference.
PrismML created and trained Bonsai. This repository contains only the OpenVINO conversion by Wondernutts. Please read and credit the original model card for the model's background and intended use.
Important: this is not the native 1-bit build
The source checkpoint expands Bonsai into ordinary FP16 tensors for compatibility with stock Hugging Face tooling. This repository quantizes those unpacked tensors to conventional OpenVINO INT4. It does not preserve PrismML's native 1-bit packing, kernels, or memory advantage.
For the smallest and fastest native Bonsai deployment, use PrismML's 1-bit MLX or 1-bit GGUF releases instead.
Conversion details
| Item | Value |
|---|---|
| Source | prism-ml/Bonsai-27B-unpacked |
| Architecture | Qwen3.5 conditional-generation/VLM graph |
| Weight compression | INT4 symmetric, group size 128 |
| Backup precision | INT8 symmetric |
| INT4 ratio | 1.0; all language-model layers requested |
| OpenVINO binary payload | approximately 13.94 GiB |
| Export stack | Optimum 2.2 development exporter, Transformers 5.2, OpenVINO IR |
The repository includes the language model, text embeddings, vision components, tokenizer, and detokenizer IRs. The advertised architectural context limit comes from the source configuration and has not been independently validated for this conversion.
Download
Download the public repository:
python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/Binary-Bonsai-27B-int4-sym-ov --local-dir ./binary-bonsai-27b-int4-ov
Run on Intel GPU
VLMPipeline is required because this export contains the conditional-generation/VLM graph. Text-only chat works without supplying an image.
import openvino_genai as ov_genai
model_dir = "./binary-bonsai-27b-int4-ov"
pipe = ov_genai.VLMPipeline(model_dir, "GPU", CACHE_DIR="./ov_cache")
config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.8
config.top_p = 0.95
pipe.start_chat("You are a concise, capable assistant.")
print(pipe.generate("Explain why the sky changes color at sunset.", generation_config=config))
pipe.finish_chat()
start_chat() uses the tokenizer/chat template shipped with the model and retains KV cache across turns. Change "GPU" to "CPU" for CPU inference. If you build a continuous-batching server on Intel GPU and encounter garbled output, disable dynamic activation quantization with DYNAMIC_QUANTIZATION_GROUP_SIZE=0 in the device/pipeline properties.
Optional image input
import numpy as np
from PIL import Image
from openvino import Tensor
image = np.asarray(Image.open("image.jpg").convert("RGB"), dtype=np.uint8)[None, ...]
print(pipe.generate("Describe this image.", images=[Tensor(image)], generation_config=config))
Vision capability and long-context behavior have not been independently benchmarked for this conversion.
License and attribution
Apache-2.0, following the source repository. Model authorship and training credit belong to PrismML; OpenVINO conversion credit belongs to Wondernutts.