QFFT-repro-S1.1-SFT-7B
This is the S1.1 / SFT baseline from QFFT: Question-Free Fine-Tuning for Adaptive Reasoning (Liu et al., 2025, arXiv:2506.12860), trained independently to reproduce the paper.
-
It is not the authors' checkpoint.
-
It is not affiliated with or endorsed by the QFFT authors, Qwen / Alibaba Cloud, or the dataset authors.
-
Code, training configs, data notices, evaluation outputs and research notes: https://github.com/RomEscape/Research-Question_Free-
-
Base model: Qwen/Qwen2.5-7B-Instruct (Apache-2.0). These weights are a full fine-tune of that model, so they are a modified version of it.
-
Companion model trained on the same responses without questions: PJiNH/QFFT-repro-S1.1-QFFT-7B
Training
| Item | Value |
|---|---|
| Method | Standard SFT. Each example is the system prompt "Please reason step by step, and put your final answer within \boxed{}.", then the question, then the response. The loss is on the response (LLaMA-Factory template qwen). |
| Data | s1_sft.json: all 1,000 questions of simplescaling/s1K-1.1 with their DeepSeek-R1 responses |
| Framework | LLaMA-Factory (bundled in the GitHub repository), full fine-tuning, bf16 |
| Hyperparameters | cutoff length 16,384; 6 epochs; learning rate 1e-5 with a cosine schedule and warmup ratio 0.1; batch size 1 with gradient accumulation 8 (effective 8); weight decay 1e-4; seed 42 |
| Hardware | 1 × NVIDIA RTX PRO 6000 Blackwell; DeepSpeed ZeRO-2 with CPU optimizer offload; SDPA attention |
| Config | LLaMA-Factory/examples/train_qfft/train_s1_sft.yaml |
| Final train loss / runtime | 0.6429 / 35,174 s |
The questions are exactly those of simplescaling/s1K-1.1 (1,000/1,000, same order). Each response is its deepseek_thinking_trajectory + "<|im_start|>answer\n" + deepseek_attempt. Six of the 1,000 responses differ from the authors' released S1_QFFT responses, in whitespace only. The hyperparameters above match the paper's Table 5, which states that SFT and QFFT share one set of hyperparameters.
Known differences from the original setup:
- Input format (our choice): SFT is the system prompt plus the question. The paper does not specify the SFT input format, and the authors released neither SFT data nor SFT checkpoints.
- Hyperparameter source: the paper's Appendix A.2 says the SFT baselines follow the official S1, LIMO and Sky-T1 hyperparameters, which contradicts Table 5. We used Table 5.
- GPU count and loss grouping: we used 1 GPU with gradient accumulation 8. The paper gives the batch size (8) but not the GPU count. The authors' released QFFT checkpoints record gradient accumulation 2.
- Attention and DeepSpeed: SDPA instead of FlashAttention-2, and DeepSpeed ZeRO-2 with CPU optimizer offload instead of ZeRO-3 from the upstream config.
- Library versions: transformers 4.52.1 and PyTorch 2.13, newer than the authors' checkpoints (transformers 4.48.2, PyTorch 2.6).
Evaluation (this reproduction)
Settings follow the released eval/eval.sh:
- Qwen chat template with the system prompt "Please reason step by step, and put your final answer within \boxed{}." The paper's Figure 8 writes
$\boxed{}$; the released code, used here, has no$. - temperature 0.6, top-p 1.0, at most 32,000 new tokens, 16 samples per problem, one vLLM 0.11.0 engine on one GPU.
The paper text states a 32,768-token budget; the released code uses 32,000. The table columns are:
- Acc: accuracy averaged over the 16 samples.
- Tokens: mean generated length.
- RAK: the paper's Definition 2, with Qwen2.5-7B-Instruct as the reference, averaged over the 16 runs.
| Benchmark | Acc | Tokens | RAK | Paper (Acc / Tokens / RAK) |
|---|---|---|---|---|
| GSM8K | 90.8 | 2.2K | 2.2 | 90.6 / 1.7K / 1.8 |
| MATH500 | 80.9 | 6.9K | 5.8 | 80.8 / 5.3K / 3.5 |
| AIME25 | 17.3 | 20.9K | 0.0 | 18.2 / 17.7K / 0.0 |
| Average | 63.0 | 10.0K | 2.7 | 63.2 / 8.2K / 1.8 |
Contamination note: GPQA
88 of the 1,000 s1K-1.1 questions have source_type Idavidrein/gpqa. This model was trained on those questions and their responses, as were the paper's S1.1 models.
Overlap with GPQA Diamond (198 items):
- 14 items share a word 8-gram with the s1K-1.1 questions or responses; 7 share a 13-gram.
- Dropping the 14 items moves this model's 4-sample GPQA accuracy from 44.3 to 43.8. LIMO-SFT, which was not trained on s1K-1.1, moves from 44.9 to 44.4.
- The gap to S1.1-QFFT stays within noise: -0.8 points [-4.9, +3.7] on all items and -1.6 [-6.1, +2.9] without the overlap.
- The method is in
scripts/gpqa_decontam.pyin the GitHub repository.
Report GPQA results for this model together with this overlap.
Usage
from vllm import LLM, SamplingParams
llm = LLM(model="PJiNH/QFFT-repro-S1.1-SFT-7B")
tok = llm.get_tokenizer()
messages = [
{"role": "system", "content": "Please reason step by step, and put your final answer within \\boxed{}."},
{"role": "user", "content": "What is the sum of the first 50 positive odd integers?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = llm.generate([prompt], SamplingParams(temperature=0.6, top_p=1.0, max_tokens=32000))
print(out[0].outputs[0].text)
License and data terms
- Weights: Apache-2.0, inherited from Qwen2.5-7B-Instruct. The
LICENSEfile in this repository carries the original notice, Copyright 2024 Alibaba Cloud. The change is the full fine-tuning described above. - Training data: simplescaling/s1K-1.1, MIT per its dataset card. The responses are DeepSeek-R1 outputs (DeepSeek-R1: MIT, Copyright (c) 2023 DeepSeek). Please cite s1 (Muennighoff et al., 2025, arXiv:2501.19393).
- Question sources with their own terms:
- 9 questions come from GAIR/OlympicArena (CC BY-NC-SA 4.0).
- 287 come from qq8933/AIME_1983_2024, whose card asks that the data not be used for training.
- Whether such data terms extend to trained weights is unresolved. These weights are released for non-commercial research.
- Full notices:
THIRD_PARTY_NOTICES.mdin the GitHub repository. The public data copy there leaves out the GPQA-sourced and OlympicArena rows.
Citation
Please cite the QFFT paper (arXiv:2506.12860) and the s1 paper (arXiv:2501.19393).