SAIL all-prompt judge (full-parameter, Qwen3.5-9B)
This model is a measuring instrument, not part of the method. It has read every prompt in ASAP 2.0 and must never select anything.
A zero-reference scorer trained on all seven prompts --- every prompt in both the training and validation halves, which is not a LOPO split --- for ten epochs, best validation QWK 0.8199. One model for the whole project rather than one per fold.
It exists for one measurement. To ask whether a simulator's essay really earns the score it was asked for, something has to read it back, and a fold's own scorer is the weakest available judge of exactly the prompt that fold holds out --- which is the prompt every generated essay answers. This model is outside the fold structure entirely.
The rule is sharp and worth restating: reporting a number it produced is legal; letting a number it produced pick a checkpoint, a reference, or a generation is a leak. Simulator epoch selection therefore continues to use each fold's own zero-reference scorer, and nothing in the pipeline reads this model's output.
What it was used for: the score-controllability table of the paper, over 14 pools of 300 essays each (seven folds x base and fine-tuned simulator), parsing 300 of 300 every time.
What these checkpoints are, and the rules they were made under
SAIL trains a simulator to write essays for a held-out ASAP 2.0 prompt at a requested holistic score, then hands those synthetic essays to a scorer as its in-context references. Both components are Qwen3.5-9B.
Four properties are not stylistic choices; breaking one invalidates every number.
- Fold k holds out prompt k+1. No essay, label or statistic of the held-out
prompt reached SFT, generation, or reference selection for that fold. A checkpoint
from
fold_0/may only be evaluated on prompt 1. - One rubric, 1--6, shared by all seven prompts. There is no score normalisation and no per-prompt range anywhere in this project.
- Grade level is a property of the prompt, not the writer. That is why conditioning on it is legal: it is known for a held-out prompt before any essay for it exists.
- Synthetic reference scores are requested scores, not human ratings. Rows in the
released pools carry
label_source='requested'. Keep that column. - Every reference must share the target essay's prompt. The scorer template states the assignment once, above the reference block, so a cross-prompt reference would be presented as though it answered the target's task.
Training recipe
Rank-32 LoRA over every linear layer of Qwen3.5-9B: 86.6M trainable parameters, 0.97% of
the 8.95B backbone. lora_alpha=64, lora_dropout=0.1, learning rate 1e-4, cosine
schedule, AdamW with weight decay 0, five epochs, bf16, SDPA, gradient checkpointing,
seed 42. Loss is taken on the completion tokens only. Because Qwen3.5 interleaves linear
attention with full attention, the adapted set includes the linear-attention projections
(in_proj_{a,b,z,qkv}, out_proj) as well as the usual attention and MLP matrices; the
target list is read off the loaded model rather than written down.
Serving
Both models load through AutoModelForCausalLM, which builds Qwen3.5's text stack
alone. vLLM cannot open these checkpoints unaided: it rewrites the unregistered
architecture name to the vision-language class, trips on the language_model. weight
prefix, asks for MRoPE, and never sizes the linear-attention state. src/scorer/ vllm_qwen35.py in the code release closes all four, and the scoring entry point pins
VLLM_ENABLE_V1_MULTIPROCESSING=0 so the registration reaches the engine rather than
dying in a spawned child.
Pinned versions: PyTorch 2.8.0, Transformers 5.17.0, TRL 1.13.0, PEFT 0.21.0 for training; a separate environment with vLLM 0.26.0 (PyTorch 2.11.0) for scoring.
Corpus and licence
ASAP 2.0, released CC BY 4.0 at scrosseye/ASAP_2.0; please cite Crossley et al., A large-scale corpus for assessing source-based writing quality: ASAP 2.0, Assessing Writing 65 (2025). Use of the base model is subject to the Qwen3.5-9B licence.