Qwen3-4B-Dolci-Think-SFT-7B
Qwen/Qwen3-4B after full-parameter supervised fine-tuning on a 600k-example subset of allenai/Dolci-Think-SFT-7B.
The model emits reasoning inside <think>...</think> blocks, as the base model and the
SFT data both do. It was trained as the starting checkpoint for research on compressing
chain-of-thought traces into short sequences of abstract-token embeddings, but nothing
about the checkpoint is specific to that use — it is an ordinary reasoning SFT of Qwen3-4B.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "sagisch/Qwen3-4B-Dolci-Think-SFT-7B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "How many r's are in strawberry?"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
Use the Qwen3 thinking-mode sampling settings — temperature 0.6, top-p 0.95, top-k 20.
Greedy decoding is explicitly discouraged for Qwen3 and tends to produce repetition
collapse. Context length is 40,960 tokens (max_position_embeddings), inherited from the
base model.
Training
| Base model | Qwen/Qwen3-4B (4B params, 36 layers) |
| Data | allenai/Dolci-Think-SFT-7B, deterministic 600k-example train subset (20k held out for eval) |
| Objective | Cross-entropy on assistant tokens only (prompt and template tokens masked) |
| Steps | 4,518 (1 epoch), ~6.0B tokens seen |
| Global batch | 128 sequences, token-budget batching at 32,768 tokens/micro-batch |
| Max sequence length | 32,768 (2,716 of 600k rows dropped as longer) |
| Optimizer | AdamW, lr 1e-5, cosine schedule, 3% warmup, β=(0.9, 0.95), wd 0.1, grad clip 1.0 |
| Precision | fp32 master weights, bf16 autocast, SDPA attention, gradient checkpointing |
| Hardware | 4× H100, DDP |
| Seed | 1337 |
Held-out eval loss over training (20k-example split, 512 examples per eval):
| Step | 10 | 3200 | 3600 | 4000 | 4400 | 4518 |
|---|---|---|---|---|---|---|
| eval loss | — | 0.8462 | 0.8450 | 0.8444 | 0.8442 | 0.8442 |
Train loss started at 1.2502 (step 10) and ended near 0.77–0.85. The curve is flat over the last ~1,000 steps, so this is a converged single epoch rather than an early stop.
These weights are the run's best checkpoint by held-out loss, cast to bf16. Optimizer
state is not included — this checkpoint is for inference and further fine-tuning from
scratch-initialized optimizer state, not for resuming the original run.
Evaluation
No benchmark numbers are reported here yet. Loss above is held-out cross-entropy on the Dolci eval split, which is a language-modeling metric, not a task-accuracy metric — do not read it as a capability comparison against the base model.
Limitations
- Single epoch on one SFT dataset, with no preference tuning or RL on top. It inherits the base model's limitations and adds Dolci-Think's distributional biases.
- No safety tuning was performed beyond whatever the base model and dataset carry.
- Not evaluated on standard reasoning benchmarks, so any claim of improvement over
Qwen/Qwen3-4Bis currently unsupported.
License
Apache-2.0, inherited from Qwen/Qwen3-4B. Dolci-Think-SFT-7B carries its own terms; see the dataset card.