ThinkingCap-Qwen3.5-2B (loss-level variant)
A merged, ready-to-serve fine-tune of Qwen3.5-2B trained with GRPO to reach the same answers with 43% fewer thinking tokens — while improving accuracy by +13 to +17 points, because the dominant failure of the base model is not verbosity but never finishing its reasoning at all.
This adapter is the loss-level variant of a controlled study on how to compress chain-of-thought at small scale. It combines three modifications to the GRPO objective — Dr.GRPO normalization, positional advantage decay inside the reasoning span, and KL restricted to that span — on top of a correctness-gated length penalty. The companion study that isolates each component is linked at the bottom.
In-domain results
Measured under a natural-length protocol: greedy decoding, a 4,096-token budget, and no forced truncation of thinking. A response whose reasoning span never closes contains no answer and is scored incorrect — so token reduction and accuracy are read off the same generations.
| Benchmark | Base Acc | Ours Acc | Base Thinking Tokens | Ours Thinking Tokens | Reduction |
|---|---|---|---|---|---|
| Held-out dev (300 problems) | 69.0 | 82.3 | 2,922 | 1,656 | ↓ 43.3% |
| GSM8K | 50.0 | 66.7 | 3,545 | 2,438 | ↓ 31.2% |
| ARC-Challenge | 70.8 | 87.5 | 2,216 | 1,247 | ↓ 43.7% |
Out-of-domain
| Benchmark | Base Acc | Ours Acc | Base Thinking Tokens | Ours Thinking Tokens | Reduction |
|---|---|---|---|---|---|
| MATH-500 | 20.8 | 20.8 | 4,096 | 3,320 | ↓ 19.0% |
Out of distribution the adapter preserves accuracy exactly while cutting a fifth of the reasoning budget. Compression transfers; the accuracy gains do not — a 2B model has little headroom on competition mathematics either way.
Failure modes
The base model's real pathology is non-termination: it finds the answer early, then loops on self-verification and never emits a closing marker.
| Metric (out-of-domain) | Base | Ours |
|---|---|---|
| Reasoning never closes within budget | 100% | 75% |
| Terminates normally (EOS) | 0% | 25% |
On in-domain data the effect is far stronger — unclosed reasoning drops from 58% to 21%. Most of the accuracy gain is simply answers that now exist.
Training
| Base model | Qwen/Qwen3.5-2B (thinking mode) |
| Method | GRPO + LoRA (rank 16, α 32, bf16), 200 optimizer steps |
| Reward | correctness-gated length penalty; zero reward when the answer is wrong or the reasoning span never closes |
| Objective modifications | Dr.GRPO normalization · positional advantage decay over the thinking span · KL restricted to the thinking span |
| Data | 10.7K verifiable problems (GSM8K + ARC training splits), deduplicated against every evaluation set |
| Hardware | 1× NVIDIA RTX 5080 (16 GB) |
No generation-time intervention is used at any point — no forced stopping, no length cap on thinking. The model learns to terminate purely from the reward.
Usage
The repository root holds the merged model — load it like any Qwen3.5 checkpoint, no PEFT required.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "drlee1/ThinkingCap-Qwen3.5-2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "If a train travels 120 km in 1.5 hours, what is its average speed?"}]
inputs = tok.apply_chat_template(
messages, add_generation_prompt=True, enable_thinking=True,
return_dict=True, return_tensors="pt",
).to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
enable_thinking=True is required — Qwen3.5-2B runs in non-thinking mode by default, and this model is trained on the thinking-mode chat template.
Honest scope
- Single training seed, 200 optimizer steps. Numbers are a screening result, not a multi-seed benchmark. Evaluation uses 300 held-out problems for the dev set and 24 items per public benchmark.
- A simpler recipe is stronger. In the companion ablation, dropping all three loss-level modifications and keeping only the reward-level length penalty reaches 79% token reduction at 90.7% dev accuracy — better on both axes than this model. Each objective modification measurably costs compression in this regime (2B, LoRA, small rollout groups). This model is released as the loss-level arm of that comparison, not as the recommended configuration.
- Findings are specific to this scale and setup; they are not a general verdict on the underlying techniques in their original settings.
Citation
@misc{reducing-think-token-2026,
title = {Reducing Thinking Tokens via Reinforcement Learning:
A Controlled Study on Loss-Level Interventions},
author = {DONGRYEOLLEE1},
year = {2026},
url = {https://github.com/DONGRYEOLLEE1/reducing-think-token}
}
Acknowledgements. Inspired by ThinkingCap-Qwen3.6-27B (BottleCapAI). Built on Qwen3.5-2B, TRL, and PEFT. The full study — including the component-wise ablation with confidence intervals — is on GitHub.