Qwen3.8-27B-OQ3 (GGUF)
[!IMPORTANT] ~5.3× smaller than the original 16-bit Qwen3.8-27B, with 95.4% accuracy retention.
OQ3 is a Post-Training Quantization (PTQ) of Qwen/Qwen3.8-27B with a calibration-based per-tensor format allocation. The name encodes the model's 3.026 bits-per-weight full-model average (body 2.967 bpw, Q2_K embedding / Q4_K output head).
Files
| file | size | full avg BPW | body BPW |
|---|---|---|---|
Qwen3.8-27B-OQ3.gguf | 10.54 GB | 3.026 | 2.967 |
mmproj-F16.gguf | 0.93 GB | — | — |
The main file embeds the model's multi-token-prediction (MTP) head (Q6_K/Q8_0, ~0.36 GB) as an optional speculative draft — llama.cpp ignores it unless you pass the MTP flags, so the live-decode footprint and the 3.026 BPW figure are unchanged. mmproj-F16.gguf is the optional vision projector for image/video input.
Benchmarks
Scores versus the original 16-bit Qwen/Qwen3.8-27B. Higher is better.
| Model | MMLU-Pro | AIME-25 | Average | Full BPW | Body BPW | Retention |
|---|---|---|---|---|---|---|
| Qwen3.8-27B (bf16) | 76.71 | 95.83 | 86.27 | 16.00 | 16.00 | 100.0% |
| OQ3 (this repo) | 73.94 | 90.42 | 82.18 | 3.026 | 2.967 | 95.4% |
| Escha W2 | 71.24 | 87.50 | 79.37 | 3.020 | 2.485 | 92.1% |
Note: The same setup was applied to all models, so scores in this table are comparable. The quantized rows are at a matched total byte budget (10.17 vs 10.15 GB of live weights). MMLU-Pro is 5-shot CoT generative over 12,032 questions; AIME-25 is 8 seeds × 30 problems with 65,536 max new tokens.
Usage (llama.cpp)
No manual download needed — llama.cpp pulls the model (and the vision projector) straight from this repo:
# OpenAI-compatible server with web UI:
llama-server -hf oracomputing/Qwen3.8-27B-OQ3-GGUF
# Chat in the terminal:
llama-cli -hf oracomputing/Qwen3.8-27B-OQ3-GGUF
Or with the llama.app launcher: llama serve -hf oracomputing/Qwen3.8-27B-OQ3-GGUF. To download the files explicitly instead:
hf download oracomputing/Qwen3.8-27B-OQ3-GGUF Qwen3.8-27B-OQ3.gguf mmproj-F16.gguf --local-dir .
then point llama.cpp at the local file with -m Qwen3.8-27B-OQ3.gguf in place of -hf ....
Faster decoding with the built-in MTP head (recommended)
Qwen3.8 ships a multi-token-prediction head, embedded in this GGUF as block 64. On llama.cpp b10502 or newer, enable it for a substantial decode speedup (community-measured +33–145% depending on hardware) — no separate draft model needed:
llama-server -hf oracomputing/Qwen3.8-27B-OQ3-GGUF \
-c 131072 -ngl 999 -fa on \
--spec-type draft-mtp --spec-draft-n-max 3 --parallel 1
--spec-draft-n-max 3 is a good default; the optimum is hardware-dependent, so try values 1–4. Measured on this file: draft acceptance ≈ 0.78. Older llama.cpp builds simply ignore the MTP tensors and run the model normally.
Vision (image / video input)
With -hf, the mmproj-F16.gguf vision projector from this repo is downloaded and paired automatically. When loading from local files, pass it explicitly:
llama-server -m Qwen3.8-27B-OQ3.gguf --mmproj mmproj-F16.gguf \
-c 131072 -ngl 999 -fa on --image-min-tokens 1024
--image-min-tokens 1024 is recommended for grounding tasks. Vision and MTP flags can be combined.
Best practices (same as Qwen/Qwen3.8-27B):
- Sampling parameters:
- Thinking mode:
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0 - Instruct (non-thinking) mode:
temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0 - If you hit endless repetition, raise
presence_penaltybetween 0 and 2 (higher values can mix languages and slightly hurt quality).
- Thinking mode:
- Adequate output length: allocate generous token budgets for reasoning-heavy tasks — up to 262,144 tokens for reasoning content and 131,072 for the final response, within the model's native 262K context.