ThinkingCap-Qwen3.8-27B — AutoRound INT4 (W4A16)
An INT4 AutoRound quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, BottleCap AI's reasoning fine-tune of Qwen/Qwen3.8-27B. It is 19.0 GB and keeps the MTP head for self-speculative decoding.
BottleCap publishes FP8, NVFP4 and GGUF builds, but no INT4 AutoRound. This one exists so the model can run on two 24 GB cards with the same tiers as the base Qwen3.8-27B in club-3090.
Quantization
| Scheme | W4A16: int4, symmetric, group size 128, auto_round:auto_gptq packing |
| Kept in BF16 | lm_head, embeddings, vision tower, GDN linear_attn.in_proj_a / in_proj_b, mtp.fc, all norms |
| MTP head | mtp.fc in BF16; the MTP layer's 7 linear weights in int4 (round-to-nearest, not tuned) |
| Tool | auto-round 0.14.2 |
| Tuning | 200 iterations, 512 samples, seqlen 2048, enable_alg_ext=True |
| Calibration | auto-round's default (NeelNanda/pile-10k) |
The recipe is also stored in club3090_build_recipe.json.
The packed layout is tensor-for-tensor identical to Frozenlock/Qwen3.8-27B-int4-AutoRound: the same quantized and BF16 module families, and the same MTP tensors. So it should load anywhere that checkpoint does. Only the tuning differs: 512 samples with alg_ext here, where that card says "default tuning".
Serving (vLLM)
vllm serve wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 \
--quantization auto_round --dtype bfloat16 --tensor-parallel-size 2 \
--speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Sampling follows the base model: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0. Thinking
effort is set with chat_template_kwargs: {"reasoning_effort": "xhigh"}, which is the template's
default.
Known upstream issues on stock vLLM, each fixed in club-3090's composes:
- MTP with prefix caching on Qwen3-Next hybrids can corrupt the recurrent state (vllm#43559; fix vllm#48375 is still open). club-3090 vendors the fix. On stock vLLM, disable prefix caching when MTP is on.
- W4A8 (
VLLM_MARLIN_INPUT_DTYPE=int8, roughly +40–50% prefill in our Ampere measurements) doesn't work on stock vLLM for AutoRound checkpoints. The setting is silently ignored (vllm#48904), and negative group scales corrupt the output (vllm#48905). Both are patched in club-3090. On stock vLLM, serve it as W4A16.
Validation
Tested on 2× RTX 3090 (PCIe, 230 W cap) with vLLM v0.30.0 through club-3090's dual-fast
configuration: TP=2, W4A8, MTP n=4, fp8 KV, 262K context. Compared against Frozenlock's INT4 of
the base Qwen3.8-27B on the same rig and image, the same day.
| this quant | |
|---|---|
| verify-full | pass |
| Decode, narrative / code (tok/s) | 74.0 / 101.0 |
| Prefill @10K / @90K (tok/s) | 1,720 / 1,246 |
| MTP acceptance length, n=4 | 3.6–5.0 (bench); 2.7–4.0 throughout the 20K run |
| 20,000-token forced generation | no drafter collapse, 0 restarts |
These are decode and serving checks. The quality evaluation, club-3090's 8-pack, is pending, and this card will be updated with it.
License
ThinkingCap is licensed under the PolyForm Small Business License 1.0.0, plus BottleCap AI's
personal-use permission; see LICENSE. This quantization is a derivative and is distributed under
the same terms. Commercial use by organizations that don't qualify as a small business needs a
commercial license from BottleCap AI.
Required Notice: Copyright 2026 BottleCap AI (https://bottlecapai.com)
ThinkingCap is derived from Qwen materials, which are licensed under Apache-2.0. See NOTICE and
LICENSE-Apache-2.0-Qwen.txt.
All credit for the model goes to BottleCap AI and the Qwen team. This repository only adds the quantization.