Model Card for Qwen3.8-27B (GGUF, with vision)
GGUF builds of Qwen3.8-27B packaged to run entirely on device, with the vision tower, the tokenizer, the merge table, the chat template and the sampling card all embedded in the single weight file. Nothing calls out to a server at inference time and there are no sibling files to keep in sync.
This repository holds the quantized weights only, not training data or the original checkpoint.
File format
.ggxf is GGUF plus the tensors and metadata keys this project adds, so these
files load in Gadeon and are not expected to load in other GGUF readers.
Files
Five points on the size/quality curve. Take ONE file, nothing beside it.
| File | Size | Tensors | Drafter | iq4_nl |
|---|---|---|---|---|
Qwen3.8-27B-IQ1_S.ggxf | 6.88 GiB | 1200 | yes | no |
Qwen3.8-27B-IQ2_XXS.ggxf | 7.88 GiB | 1200 | yes | no |
Qwen3.8-27B-IQ3_XXS.ggxf | 11.05 GiB | 1200 | yes | no |
Qwen3.8-27B-IQ4_XS.ggxf | 14.14 GiB | 1200 | yes | yes |
Qwen3.8-27B-Q4_K_S.ggxf | 15.17 GiB | 1200 | yes | yes |
Every file carries the vision tower, the tokenizer, the chat template and the
sampling card. The tower travels inside the GGUF rather than as a separate
mmproj, and the sampling parameters travel with it, so a runtime never has
to guess them or borrow another model's settings.
The iq4_nl column matters for third-party runtimes: every block type here is
a standard ggml one inherited from the upstream quantization, but the two
widest mixes include iq4_nl, which some runtimes decode and others do not.
The additions are the embedded vision tower, the drafter where the trunk lacked one, and the metadata. Those have only been exercised in the companion on-device runtime -- third-party tools have not been tested against these files, and a tool that does not expect a tower inside the weight file may ignore it or refuse it.
Provenance
Each language trunk is one of unsloth's
quantizations of
Qwen3.8-27B -- UD-IQ1_S, UD-IQ2_XXS, UD-IQ3_XXS, UD-IQ4_XS and UD-Q4_K_S --
used as published. What this build adds is the vision tower and the metadata:
the v.* / mm.* tensors and clip.* configuration, the generation config
with its sampling matrix and its stop-token list. The tower is carried at
source precision (F16/F32) and is bit-identical to the tower in the origin
checkpoint.
The UD-IQ3_XXS, UD-IQ4_XS and UD-Q4_K_S trunks ship unsloth's nextn layer
already. UD-IQ2_XXS and UD-IQ1_S do not, so their drafter block was taken
verbatim from unsloth's separate
MTP/mtp-Qwen3.8-27B-Q4_0.gguf.
Model Details
Model Description
Qwen3.8-27B is a dense hybrid. Its layers alternate three Gated DeltaNet linear-attention blocks to one full attention block. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall.
These builds are multimodal. The vision tower is included, so image input works from any one of these files alone.
Every build carries a multi-token-prediction drafter. Speculative decode over it was measured token-identical to plain greedy on this repository's own runtime; a runtime that does not speculate decodes normally.
- Developed by: Qwen, Alibaba (base model); quantization by unsloth; packaging by leok7v
- Model type: Hybrid Gated DeltaNet + attention causal language model with a vision encoder
- Language(s): English and the languages of the base model
- License: Apache 2.0, inherited from the base model
- Quantized from model: Qwen/Qwen3.8-27B
Model Sources
- Repository (this build): https://huggingface.co/leok7v/Qwen3.8-27B
- Base model: Qwen/Qwen3.8-27B
- Trunk quantization: unsloth/Qwen3.8-27B-GGUF
Uses
Direct Use
On-device chat, summarization, drafting, question answering over supplied text, and description of supplied images, in offline or privacy-sensitive settings where prompts must not leave the device.
Out-of-Scope Use
Anything the base model's licence and card exclude. These builds add no alignment, no safety tuning and no filtering of their own; they are the upstream model in a different numeric format. Not a basis for legal, medical, financial or safety decisions, and not intended for high-stakes automated decisions without human review.
Bias, Risks, and Limitations
The base model's biases and failure modes carry over unchanged.
Quantization damage is real and it is not uniform across these files. The 1-bit build is the most heavily compressed in this family and is noticeably further from the upstream model than the wider ones: outputs will not match the upstream model token for token, the gap is widest where the model was already uncertain, and short factual questions can be answered incorrectly or declined even when the model "knows" the answer at higher precision. The 2-bit build is better but still visibly damaged. Long multi-step derivations are where errors accumulate fastest in every case.
Anyone depending on one of these builds should evaluate it on their own task rather than assume it behaves like the full-precision model, or like the other files here. If accuracy matters more than footprint, prefer a wider quant.
Recommendations
Treat generations as drafts to verify, not as fact. For anything where accuracy matters, and for arithmetic in particular, check the result.
How to Get Started with the Model
The tokenizer, the chat template and the sampling card are embedded, so a runtime reads all of them from the GGUF. The chat template documents the roles, the thinking block and the tool-call format.
The base model card recommends these sampling parameters, and the embedded sampling card carries them:
| mode | temperature | top_p | top_k | presence_penalty |
|---|---|---|---|---|
| Thinking | 1.0 | 0.95 | 20 | 0.0 |
| Instruct (non-thinking) | 0.7 | 0.80 | 20 | 1.5 |
reasoning_effort is supported by the template; low and medium cost fewer
tokens per turn than the default.
Generation stops on either <|im_end|> (248046) or <|endoftext|> (248044).
Both ids are carried in the embedded generation config and in
tokenizer.ggml.eos_token_ids; a runtime that reads only the scalar
tokenizer.ggml.eos_token_id will see <|im_end|> alone.
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 5120 |
| Layers | 64 (Gated DeltaNet + attention, three to one) |
| Attention heads | 24 query / 4 key-value (grouped-query) |
| Head dimension | 256 (partial rotary, 64) |
| Linear-attention state | 128, 16 groups, inner size 6144 |
| Feed-forward | 17408 |
| Vocabulary | 248320 |
| Context length | up to 262144 tokens |
| Vision encoder | 27 blocks, 768px tiles, 2x2 spatial merge |
Weight format
GGUF v3. Every trunk is a MIXED-precision assignment rather than one block type throughout: the widths are chosen per tensor, so the most sensitive rows keep more bits than the bulk of the model. The name on the file is the dominant type, not the only one.
Dominant trunk types per file, by tensor count:
| File | Leading types |
|---|---|
| IQ1_S | IQ1_S 264, Q8_0 96, IQ2_XXS 59, IQ1_M 25, IQ2_XS 21 |
| IQ2_XXS | IQ2_XXS 143, IQ1_S 100, Q8_0 96, IQ3_XXS 40, IQ2_S 36 |
| IQ3_XXS | IQ3_XXS 120, IQ3_S 106, Q8_0 98, IQ4_XS 45, IQ2_S 35 |
| IQ4_XS | IQ4_XS 211, Q8_0 98, Q4_K 55, Q5_K 51, IQ3_S 46 |
| Q4_K_S | IQ4_XS 172, Q8_0 99, Q4_K 95, Q5_K 80, Q6_K 18 |
The Q4_K_S mix is worth a second look: despite the name, more of its trunk is
IQ4_XS than Q4_K, and it is the file that carries the most iq4_nl.
In every file the norms, layer scalars and recurrent-state parameters stay F32 (360 tensors), and the 334 vision-tower tensors stay at source precision (222 F32 + 112 F16).
Citation
Please cite the upstream Qwen3.8-27B model. See the base model card at Qwen/Qwen3.8-27B for the canonical reference and license.
Model Card Contact
leok7v on Hugging Face.