CalibrateQwen Unified
CalibrateQwen Unified is an on-policy distilled Qwen/Qwen3.5-4B checkpoint for multiple-choice reasoning with structured confidence and abstention. We selected this checkpoint from a matched comparison against the base model and an off-policy distilled checkpoint.
Release format
- Base model:
Qwen/Qwen3.5-4B - Export format:
merged - Training method: on-policy teacher-student distillation
- Source checkpoint:
tinker://e45e676a-8586-5d5d-a00f-2c4527c33929:train:0/sampler_weights/final - Renderer:
qwen3_5_disable_thinking - Dataset:
ritwikraha/calibrate-qwen-curated - License: Apache-2.0, inherited from the base model
Why this checkpoint
The selected checkpoint delivered the strongest general-purpose result in the core report. The primary in-domain and OOD splits each contain 250 evaluated examples. These are release-candidate measurements rather than complete benchmark estimates when the report scope is core.
| Metric | Base | Off-policy | Unified on-policy |
|---|---|---|---|
| In-domain accuracy | 0.8800 | 0.8720 | 0.8840 |
| Structured-output validity | 0.7080 | 0.8400 | 0.9320 |
| Verbal-confidence ECE | 0.2314 | 0.2232 | 0.1182 |
| Multiclass Brier score | 0.1891 | 0.2000 | 0.1831 |
| Multiclass NLL | 0.3811 | 0.4051 | 0.3686 |
| Selective accuracy | 0.9593 | 0.9497 | 0.9674 |
OOD accuracy is 0.7600, OOD multiclass NLL is 0.8066, and OOD AURC is 0.1014. Option-order choice-text consistency is 0.8940.
Output contract
Use the following system instruction:
Answer the multiple-choice question. Return one JSON object with exactly the keys "answer", "confidence", "justification", and "abstain". answer must be one uppercase option label from the prompt. confidence must be a number between 0 and 1. justification must be one concise sentence. abstain must be a JSON boolean.
Expected response:
{"answer":"B","confidence":0.91,"justification":"The evidence directly supports option B.","abstain":false}
Loading the model
For a merged release:
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ritwikraha/calibrate-qwen-unified")
model = AutoModelForCausalLM.from_pretrained(
"ritwikraha/calibrate-qwen-unified",
device_map="auto",
torch_dtype="auto",
trust_remote_code=True,
)
For the PEFT adapter release:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", device_map="auto")
model = PeftModel.from_pretrained(base, "ritwikraha/calibrate-qwen-unified-adapter")
Serving
Merged model:
vllm serve ritwikraha/calibrate-qwen-unified
PEFT adapter:
vllm serve Qwen/Qwen3.5-4B --enable-lora --lora-modules calibrate=ritwikraha/calibrate-qwen-unified-adapter
Calibration policy
The validation-fitted answer-probability temperature is 1.082009 and the 80 percent target-coverage threshold is 0.890585. Load calibration_config.json and use calibration.py when the serving stack scores every option label. The generated verbal confidence remains available when option scoring is unavailable.
Limitations
The release targets English multiple-choice prompts. Calibration was fitted on the curated validation mixture and can shift under new domains, prompt templates, quantization, or decoding settings. The provided evaluation snapshot uses 250 examples per primary split. Refit calibration after material deployment changes and use a full evaluation before making final benchmark claims.
Reproducibility
Training, dataset curation, evaluation, and publication notebooks are available in AutoRegressive-Bhasha. The model repository includes the frozen calibration policy, output schema, evaluation summary, and compact report evidence.