Qwen3.5-9B-GGUF (OraQuant)
This repository contains GGUF builds of Qwen3.5-9B, quantized by Ora Computing with OraQuant (OQ) - Ora Computing's proprietary calibrated quantization.
These are llama.cpp-compatible quantizations of Qwen/Qwen3.5-9B; the underlying weights are unchanged Qwen3.5-9B weights at reduced precision.
Text only.
Qwen/Qwen3.5-9Bis a multimodal model; these GGUFs contain only the language model (text input -> text output). The vision/video input encoders are not included.
Model Overview
- Model name: Qwen3.5-9B-GGUF (OraQuant)
- Base model:
Qwen/Qwen3.5-9B(Apache-2.0, Alibaba Cloud) - these are GGUF quantizations of it - Parameters: ~9 billion (unchanged from the base model)
- Quantization: OraQuant (OQ) mixed-precision GGUFs produced by Ora Computing, provided in three footprints -
OQ-Q4_K_M(higher quality),OQ-Q3_K_M(smaller/faster), andOQ-IQ2_XXS(smallest, mixed i-/k-quant). - Not fine-tuned, not parameter-reduced: the model architecture and parameter count are identical to the base model; only the weight precision is reduced.
- Purpose: Evaluation/test-use only; optimized for local/offline inference and internal benchmarking.
- License: See
LICENSE(Custom Model License Agreement).
Files in this repo
| File | What it is | Size |
|---|---|---|
Qwen3.5-9B-OQ-Q4_K_M.gguf | Language model, OraQuant Q4_K_M (higher quality) | ~5.7 GB |
Qwen3.5-9B-OQ-Q3_K_M.gguf | Language model, OraQuant Q3_K_M (smaller/faster) | ~4.7 GB |
Qwen3.5-9B-OQ-IQ2_XXS.gguf | Language model, OraQuant IQ2_XXS-class (smallest, mixed i-/k-quant) | ~3.2 GB |
LICENSE | Custom Model License Agreement | - |
Usage
These GGUFs load with stock upstream llama.cpp (no patch required); use a build with Qwen3.5 support.
export MODEL=/path/to/Qwen3.5-9B-OQ-Q4_K_M.gguf # or the Q3_K_M / IQ2_XXS file
Interactive chat:
./build/bin/llama-cli -m "$MODEL" -ngl 99
Single-shot completion (-st runs one turn then exits):
./build/bin/llama-cli -m "$MODEL" -ngl 99 -st -p "Explain the Chudnovsky algorithm in two sentences."
OpenAI-compatible server (Web UI at http://localhost:8080):
./build/bin/llama-server -m "$MODEL" -ngl 99 \
--alias qwen3.5-9b --host 0.0.0.0 --port 8080
Qwen3.5 is a reasoning model; the chat template and thinking behaviour are carried in the GGUF.
For best results use the upstream-recommended sampling settings, e.g.
--temp 0.6 --top-p 0.95 --top-k 20 --presence-penalty 1.0. Long thinking phases are
normal on hard prompts, especially for the smallest (OQ-IQ2_XXS) footprint.
Intended Use & Restrictions
Permitted use
- Internal testing, benchmarking, and evaluation of the model by the named Licensee.
- Exploration of model behaviours, prompt engineering, and non-production prototypes.
Prohibited use
- Deployment in a production or commercial service, publicly-facing API, resale, or redistribution.
- Fine-tuning or creating derivative models for production use without a separate agreement.
- Reverse-engineering the quantization/calibration used to produce these files.
- Disclosure or sharing of the model (or its weights) to third parties beyond the named Licensee.
Out-of-scope use
- Use in regulated or safety-critical contexts (unless separately permitted).
- Any use that violates the Apache License, Version 2.0 under which the upstream model is distributed.
Quantization
- Method: OraQuant (OQ), Ora Computing's proprietary calibrated quantization. The released files are mixed-precision GGUFs (K-quants; the
OQ-IQ2_XXSfile additionally uses i-quants). - No fine-tuning: the weights are the original
Qwen/Qwen3.5-9Bweights; no additional training was performed. - No parameter-count change: the architecture and ~9B parameter count are unchanged; only weight precision is reduced.
- Footprints:
OQ-Q4_K_Mfor higher quality,OQ-Q3_K_Mfor a smaller/faster footprint,OQ-IQ2_XXSfor the smallest footprint (expect visible quality loss on the hardest long-reasoning tasks at this size).
Limitations & Risks
- Quantized models may not replicate the full behaviour of the base model under all prompt categories, particularly domain-specific or rare inputs.
- The model is provided as-is for testing only and is not certified for production use.
- Users should validate outputs carefully and monitor for bias or unintended behaviours.
Upstream Attribution
This model is derived from the Qwen3.5-9B model released by Alibaba Cloud under the Apache License, Version 2.0.
"Copyright 2025 Alibaba Cloud. Licensed under the Apache License, Version 2.0."
For full terms, see: https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0
Contact & Support
For licensing inquiries or to request extended evaluation rights, please contact: info@oracomputing.com
Repository and model access are regulated. Do not redistribute or share without explicit written permission from Ora Computing.