Qwen3.6-27B ExpThink — step 60
This is a text-only efficient-reasoning fine-tune of
Qwen/Qwen3.6-27B. It reduces the
number of tokens emitted inside the model's thinking span while retaining the
base model's observed accuracy on the fixed evaluation described below.
This is a continuation of
ryankim17920/qwen3p6-27b-expthink-step50,
trained for an additional 10 steps (steps 51–60) with a fresh optimizer and
scheduler. The fresh-optimizer caveat applies: the continuation was not a
seamless extension of the step-50 training state.
This repository contains only the tuned language-model tower
(Qwen3_5ForCausalLM). It does not contain the base model's vision tower and
must not be advertised or loaded as an image-text model. The Apache-2.0 license
is inherited from the base model.
Results
The paired evaluation used decoding seeds 0, 1, and 2, temperature 1.0, top-p 0.95, top-k 20, thinking enabled, and at most 200 examples per dataset. IID generations were capped at 8,192 tokens and OOD generations at 32,768.
| split | base accuracy | tuned accuracy | base thinking tokens | tuned thinking tokens | reduction |
|---|---|---|---|---|---|
| IID | 94.74% | 94.86% | 1,071.4 | 506.7 | 52.71% |
| OOD | 83.95% | 86.45% | 6,231 | 4,125 | 33.80% |
Truncation fell from 0.45% to 0.33% on IID and from 5.96% to 2.56% on OOD.
GPQA Diamond truncation fell from 14.65% to 6.73%.
Three decoding seeds at this sample size do not establish a statistically
conclusive accuracy improvement; the supported conclusion is no observed
accuracy regression. Aggregate evidence and full protocol metadata are in
{{EVAL_REPO}}.
Compared to step 50 (IID 46.75% token reduction, OOD 28.81%), step 60 improves compression on both splits (IID 52.71%, OOD 33.80%) with a small IID accuracy delta of −0.21 pp relative to step 50 and an OOD accuracy improvement of +0.22 pp.
Training
The model was trained for 10 additional successful updates (steps 51–60) over a
7,176-prompt draw from the same mixture as step 50: GSM8K, ARC Challenge, ARC
Easy, CommonsenseQA, OpenBookQA, QASC, and SciQ. Each sampled training prompt
produced eight rollouts. Training used a correctness-gated ExpThink reward that
compares a correct response with the shortest previously observed correct response
for the same prompt; wrong and truncated responses receive no compression reward.
The continuation used no KL reward term and a learning rate of 1e-6. The
optimizer and scheduler were reset at step 51 (fresh start, not a resume of
step-50 state). Final-step training health: entropy 0.2614 (within [0.18, 0.40]),
mean reward 0.5435, rollout/actor correlation 0.9990, clip fraction 0.0%,
aborted ratio 0.0%.
Only step 60 is published.
Use
Use a recent Transformers release with Qwen3.5 text-model support. The exported configuration records Transformers 5.6.0; Transformers 5.7 or newer is the conservative requirement for downstream use. The architecture uses gated linear attention, so accelerator-specific kernels may also be required by the chosen runtime.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ryankim17920/qwen3p6-27b-expthink-step60"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
torch_dtype="auto",
device_map="auto",
)
messages = [{"role": "user", "content": "Solve 17 * 23. End with Answer: N"}]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
enable_thinking=True,
return_tensors="pt",
).to(model.device)
output = model.generate(inputs, max_new_tokens=8192)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
The evaluation used a temporary composite containing this exact tuned text tower plus unchanged base-only multimodal tensors because that was the proven tensor-parallel SGLang serving path. The released artifact is the independently useful text tower, not that temporary composite.
Limitations
- The evaluation covers English verifiable QA and multiple-choice tasks, not general chat, coding, multilingual use, tool use, or safety behavior.
- Results use up to 200 examples per dataset and three decoding seeds.
- Shorter reasoning is not a correctness guarantee. Users should verify answers in high-impact settings.
- This fine-tune did not add a safety stage. It inherits base-model risks and may produce inaccurate, biased, or harmful text.
- The model is text-only despite its multimodal base model.
- This is a continuation from step 50 with a fresh optimizer and scheduler; it is not a seamless extension of the step-50 training trajectory.
Reproducibility
Training and evaluation code is maintained in
RyanKim17920/open-thinkingcap.
The evaluation artifact repository records the dataset source manifest, model
fingerprints, sampling settings, engine version, tensor-parallel size, and
confidence intervals.
publication_manifest.json records every released file's byte size, full
SHA-256, and Git blob OID. Publication verifies the Hub's LFS SHA-256 or Git blob
OID for each file before the repositories receive their final visibility.