Ornith-1.5-9B — INT4 AutoRound
INT4/W4A16 AutoRound quantization of ornith-ai/Ornith-1.5-9B.
Quantization
- AutoRound: 0.14.2
- Weight format: INT4 / W4A16
- Group size: 128
- Symmetric quantization
- Packing format:
auto_round:auto_gptq - Calibration dataset:
NeelNanda/pile-10k - Calibration samples: 128
- Calibration sequence length: 512
- Model type: Qwen3.5 multimodal conditional generation
The language-model layers are quantized to INT4. The visual tower and other unsupported non-text layers remain in their original higher-precision format.
Transformers usage
from transformers import AutoProcessor, AutoModelForImageTextToText
model_id = "letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
inputs = processor(
text="Explain quantization in one sentence.",
return_tensors="pt",
)
inputs = {k: v.to(model.device) if hasattr(v, "to") else v for k, v in inputs.items()}
output = model.generate(**inputs, max_new_tokens=64)
print(processor.batch_decode(output, skip_special_tokens=True)[0])
Standard vLLM compatibility
The language-model values were not re-quantized. For standard vLLM deployment, use the compatibility-repacked checkpoint layout generated from this artifact. The repackaging normalizes duplicated tensor-name prefixes and tokenizer metadata, keeps the language model INT4/W4A16, and restores the vision tower in its original higher-precision format. Vision input remains supported.
Example deployment arguments:
vllm serve letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound --tensor-parallel-size 2 --trust-remote-code --quantization auto_round --served-model-name Ornith-1.5-9B-INT4-W4A16-AutoRound
Verification
The artifact was verified with Transformers and standard vLLM. Text generation and an image-input request both completed successfully with the compatibility-repacked layout.
Attribution and license
This is a derived quantized artifact of ornith-ai/Ornith-1.5-9B. Please follow the upstream model's MIT license and review the upstream model card for intended use, limitations, and full attribution.