👉 Newer release: ThinkingCap-Qwen3.8-27B-GGUF — GGUF builds of ThinkingCap-Qwen3.8-27B, the second model in the ThinkingCap series, finetuned from Qwen3.8-27B. Read the blog post. Other quantized builds: FP8 · NVFP4 · NVFP4A4-AWQ · MLX 4-bit DWQ. 👈

bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. ≈4.7 bits per weight for Q4_K_M instead of the 16-bit f16 source — cutting download size and memory severalfold at a small, measured quality cost.

Files

f16 is the unquantized source; Q8_0 is near-lossless; Q6_K is a slightly smaller near-lossless option; Q4_K_M is the recommended size/quality balance for most local setups.

Usage (llama.cpp)

# pull a specific quant straight from the Hub and chat
llama-cli -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M -p "Hi"

# or download one file and run it
huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf --local-dir .
llama-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "Hi"

Speculative decoding (MTP)

llama.cpp can run MTP (multi-token-prediction) self-speculative decoding on these GGUFs for a decode speed-up — no separate draft model needed. Add --spec-type draft-mtp when serving:

llama-server -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M --spec-type draft-mtp

Set the draft length with --spec-draft-n-max (e.g. 4). Requires a recent llama.cpp build with MTP support.

Vision (image input)

ThinkingCap is a vision-language model. Image input needs the multimodal projector mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf (in this repo) loaded alongside a text GGUF — the single f16 mmproj pairs with any of the quants above.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio auto-detects it and enables the image (🖼️) button.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF \
  ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --image photo.jpg -p "Describe this image."

• llama-server: add --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf to expose an OpenAI-compatible vision endpoint.

Expected performance

Measured on our serving harness (llama.cpp, CUDA llama-server) over MMLU-Pro (reasoning) and RealWorldQA (vision), N=200/dataset × 3 seeds, sampled decoding (temp 1.0 / top_p 0.95 / top_k 20). acc is the mean ± 95% CI across seeds; task s / tok/s are measured per request during the eval at batch size 16 (a throughput setting — a single local user decoding one request at a time will see meaningfully higher tok/s, but the per-row comparison is unaffected). For the full multi-benchmark evaluation see the main model card.

Accuracy is statistically identical across f16, the quantized variants, and the base model — every 95% CI overlaps (MMLU-Pro ≈0.89–0.91, RealWorldQA ≈0.78–0.82). Quantization is near-lossless, and the finetune matches base accuracy. What these GGUFs buy you is brevity → speed: the finetune reasons ~2× shorter than base (≈1000 vs ≈2100 MMLU-Pro tokens; ≈330 vs ≈700 RealWorldQA), so at equal accuracy it finishes each task far faster. MTP self-speculative decoding (--spec-type draft-mtp, n=4) accepts ≈3.3–3.7 tokens per verify step on top. The bolded row — Q4_K_M + MTP — is the fastest per task and the recommended size/quality balance; Q8_0 and Q6_K are near-lossless alternatives at larger sizes. For reference we also list unsloth's Dynamic GGUFs of the base model (UD-*): same llama.cpp path and same accuracy, but base-model quants reason ≈2× longer (none of the finetune's token savings), so they finish only ~1.2–2.2× faster than base-std vs our ~2.6–3.9×.

median tokens = median completion length; task s = measured end-to-end wall-clock per request (bs=16); tok/s = per-request decode rate under that load; speedup = base·bf16·std task s ÷ row task s — same llama.cpp path for every row, so the within-table comparison is apples-to-apples.

MMLU-Pro (reasoning)

RealWorldQA (vision)