qwen-3.5-27b-absorb
qwen-3.5-27b-absorb is Qwen/Qwen3.5-27B trained with Absorb, the
primitive-privileged self-distillation method from The Missing Primitive: Diagnosing and
Repairing Mathematical Reasoning in Large Language Models.
- A Mathematical Primitive is the essential conceptual observation that reveals why a problem can be solved.
- Teacher and student are the same model. The teacher sees the problem together with its primitive as privileged information; the student sees only the problem and generates its own solution. A bounded override transfers the teacher's guidance along the student's own trajectory.
- At test time the model gets no primitive; it is a drop-in replacement for the base model.
- Training data: shuoxing/math-phd-qual-709, 709 mathematics Ph.D. qualifying-exam proof problems. No final answers or reward model are used.
Code, data, and full instructions: https://github.com/taco-group/Math-Primitive.
Model details
| Base model | Qwen/Qwen3.5-27B @ fc05daec18b0a78c049392ed2e771dde82bdf654 |
| Architecture | Qwen3_5ForConditionalGeneration (same layout as the base; only the language tower changed) |
| Adapter | LoRA r=64, alpha=128, on q/k/v/o_proj, gate/up/down_proj, in_proj_qkv/a/b/z, out_proj, merged into the weights |
| Objective | bounded override: reverse KL on the teacher's top-128 support with a one-sided clamp τ = 0.06; teacher = base weights (adapter disabled) |
| Privileged information | the problem's one-sentence primitive (core_concept, median 18 words) |
| Optimization | fused AdamW, no weight decay, lr 5e-6 linear schedule, effective batch 32, gradient clipping 0.1, seed 42, bf16 |
| Rollouts | 1 on-policy rollout per problem, temperature 1.0, top-p 0.95, top-k 20, max 4096 new tokens |
| Epochs | 1 |
The vision tower is copied unchanged from the base checkpoint. Training and evaluation are text-only.
Usage
Serve with vLLM (add --tensor-parallel-size 2 if the model does not fit on one GPU). Do not pass --reasoning-parser: our evaluation reads the full
thinking trace plus answer from message.content.
vllm serve shuoxing/qwen-3.5-27b-absorb --served-model-name qwen-3.5-27b-absorb \
--max-model-len 131072 --gpu-memory-utilization 0.9
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="qwen-3.5-27b-absorb",
messages=[{"role": "user", "content": "Prove that there are infinitely many primes."}],
max_completion_tokens=32768,
)
print(resp.choices[0].message.content)
Sampling defaults: The base checkpoint's generation_config.json (temperature 0.6, top-p 0.95, top-k 20) is kept unchanged. It is the default for any request that does not set these fields, exactly as in our evaluation.
With transformers, AutoModelForCausalLM.from_pretrained("shuoxing/qwen-3.5-27b-absorb") loads the
text-only language model.
Evaluation
The model is evaluated with no primitive at test time. Inference and scoring scripts for the Prim benchmark (shuoxing/Prim) are in the GitHub repository for all four dimensions: Generation (solve the problem directly), Discovery (write the primitive from the problem alone), Digestion (write the primitive from the problem and a correct solution), and Execution (solve given the gold primitive), with the exact token budgets and context lengths.