Llama.cpp GGUF quantizations of ThinkingCap-Qwen3.6-27B-heretic
Using llama.cpp release b10068 for quantization. All K-quants and non-imatrix quants are converted directly from f16; all IQ-quants use the included imatrix.gguf, calibrated on 580 chunks of wikitext-2-raw-v1.
Original model: egibson/ThinkingCap-Qwen3.6-27B-heretic, a decensored (abliterated) version of bottlecapai/ThinkingCap-Qwen3.6-27B produced with Heretic.
Note: this model is exported with the multi-token-prediction (MTP) head excluded (
--no-mtp). The base architecture's GGUF converter has an unresolved bug where the MTP head's metadata is written but its tensors are not, producing GGUF files that fail to load. Excluding it here produces a complete, correctly-loading model — MTP is a speculative-decoding speed optimization, not something the model needs to function.
Files
| Filename | Quant type | File Size | Description |
|---|---|---|---|
| ThinkingCap-Qwen3.6-27B-heretic-f16.gguf | f16 | 50.1GB | Full 16-bit precision. Lossless conversion from the original weights; largest file, use as a source for further requantization or when you have the RAM/VRAM and want maximum fidelity. |
| ThinkingCap-Qwen3.6-27B-heretic-Q8_0.gguf | Q8_0 | 26.6GB | 8.5 bpw. Essentially indistinguishable from f16 in practice; the safest choice if size isn't a major constraint. |
| ThinkingCap-Qwen3.6-27B-heretic-Q6_K.gguf | Q6_K | 20.6GB | 6.57 bpw. Very high quality, very close to f16, noticeably smaller than Q8_0. |
| ThinkingCap-Qwen3.6-27B-heretic-Q5_K_M.gguf | Q5_K_M | 17.9GB | 5.72 bpw. High quality, a good middle ground if Q6_K is too large. |
| ThinkingCap-Qwen3.6-27B-heretic-Q5_K_S.gguf | Q5_K_S | 17.4GB | 5.55 bpw. Slightly smaller than Q5_K_M with a small further quality trade-off. |
| ThinkingCap-Qwen3.6-27B-heretic-IQ4_XS.gguf | IQ4_XS | 14.0GB | 4.48 bpw, imatrix-calibrated. Usually beats Q4_K_S/Q4_0 at a similar or smaller size thanks to importance-matrix guidance — a strong pick if your loader supports IQ-quants. |
| ThinkingCap-Qwen3.6-27B-heretic-Q4_K_M.gguf | Q4_K_M | 15.4GB | 4.92 bpw. Default recommendation for most use cases — good balance of quality and size, widely compatible. |
| ThinkingCap-Qwen3.6-27B-heretic-Q4_K_S.gguf | Q4_K_S | 14.5GB | 4.63 bpw. Smaller than Q4_K_M with slightly lower quality; prefer IQ4_XS instead if your setup supports it. |
| ThinkingCap-Qwen3.6-27B-heretic-Q4_0.gguf | Q4_0 | 14.4GB | 4.60 bpw. Legacy quant format (no k-quant mixing); mainly useful for older/ARM-optimized runtimes that specifically target Q4_0. |
| ThinkingCap-Qwen3.6-27B-heretic-Q3_K_L.gguf | Q3_K_L | 13.4GB | 4.26 bpw. The largest/highest-quality of the Q3_K family. |
| ThinkingCap-Qwen3.6-27B-heretic-IQ3_M.gguf | IQ3_M | 11.7GB | 3.74 bpw, imatrix-calibrated. Good quality-per-bit in the ~3-3.5bpw range. |
| ThinkingCap-Qwen3.6-27B-heretic-Q3_K_M.gguf | Q3_K_M | 12.4GB | 3.95 bpw. Noticeable quality loss begins here; usable for memory-constrained setups. |
| ThinkingCap-Qwen3.6-27B-heretic-IQ3_XS.gguf | IQ3_XS | 11.1GB | 3.56 bpw, imatrix-calibrated. Smaller than Q3_K_M at comparable or better quality. |
| ThinkingCap-Qwen3.6-27B-heretic-Q3_K_S.gguf | Q3_K_S | 11.2GB | 3.59 bpw. Not recommended unless you specifically need a non-IQ 3-bit quant; IQ3_XS is smaller and generally better. |
| ThinkingCap-Qwen3.6-27B-heretic-Q2_K.gguf | Q2_K | 10.0GB | 3.18 bpw. Significant quality loss; only for very constrained hardware. |
| ThinkingCap-Qwen3.6-27B-heretic-IQ2_M.gguf | IQ2_M | 9.3GB | 2.97 bpw, imatrix-calibrated. Relatively usable for a ~3bpw quant thanks to imatrix guidance, but still noticeably lossy — a 27B model at this size will generally beat a much smaller model at higher precision, but expect real degradation. |
| ThinkingCap-Qwen3.6-27B-heretic-IQ2_S.gguf | IQ2_S | 8.7GB | 2.78 bpw, imatrix-calibrated. Smallest quant offered here. Extreme compression — only worth it if nothing larger fits your hardware at all. |
| mmproj-ThinkingCap-Qwen3.6-27B-heretic-f16.gguf | f16 (vision projector) | 0.9GB | Required alongside any text-model file above for image/video input. |
| imatrix.gguf | — | 13MB | Importance matrix used for the IQ-quants above, calibrated on wikitext-2-raw-v1 (580 chunks). Provided so you can produce your own custom quants without recomputing it. |
Usage
This is a multimodal model — pass the mmproj file alongside any text-model file to enable image/video input:
llama-server -m ThinkingCap-Qwen3.6-27B-heretic-Q4_K_M.gguf --mmproj mmproj-ThinkingCap-Qwen3.6-27B-heretic-f16.gguf
Text-only usage works without --mmproj.
Downloading using huggingface-cli
hf download egibson/ThinkingCap-Qwen3.6-27B-heretic-GGUF --include "ThinkingCap-Qwen3.6-27B-heretic-Q4_K_M.gguf" "mmproj-ThinkingCap-Qwen3.6-27B-heretic-f16.gguf" --local-dir ./
Which file should I choose?
If you're not sure, start with Q4_K_M — the best balance of quality and size for most hardware, and the most widely compatible format. If your runtime supports IQ-quants, IQ4_XS is a strong alternative at a similar size with imatrix-guided quality. Move up to Q5/Q6/Q8/f16 if you have the RAM/VRAM to spare, or down to the 3-bit/2-bit tiers only if nothing larger fits.