goff Lite
A standalone, single-label scoring head over microsoft/harrier-oss-v1-0.6b, fine-tuned
on typed decisions and merged so vLLM can serve it without a PEFT adapter at runtime.
The model scores one candidate at a time. A question's candidates are each rendered into their own prompt, scored independently, and softmaxed together within the question to give a distribution over candidates. It is not a chat model and does not generate text.
- Architecture:
Qwen3ForSequenceClassification, 28 layers, hidden size 1024, 1 output label (problem_type: regression) - Precision: bfloat16 (~1.19 GB)
- Context window: 32,768 tokens (inherited from the base model). Training prompts were truncated at 1024.
Training
Fine-tuned from microsoft/harrier-oss-v1-0.6b with
FuRA (Full-Rank Adaptation with spectral
preconditioning) on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and
down_proj, full per-block rank, output_one_block factorization, SVD conversion, with
the score head trained alongside via modules_to_save.
Objective is a listwise RLCD recipe: soft-target cross entropy (weight 1.0), noisy-logit
policy gradient with 4 samples per question, log and spherical proper-scoring rewards
(spherical weight 0.75), and ranked probability score for ordinal score questions
(weight 1.0). Exploration sigma annealed 0.4 → 0.1.
Data
23,526 typed questions from three sources, split into 22,350 train / 1,176 validation by case, so questions that share one input state never straddle the split:
| Source | Questions | Notes |
|---|---|---|
LocalLLaMA/typed-decisions | 6,000 | all config, train split; 1,200 cases × 5 questions, all soft-labelled |
jaredpalmer/kev eval suites | 7,222 | the train splits checked into the repo at commit 90990a5, converted to one question per record. Not the jaredpalmer/kev-suites Hub mirror, which holds the larger training pool. 24 sources: banking77, AG News, Yelp, BoolQ, MNLI, SST-5, TREC, DBpedia, IMDB, plus kev's generated policy, contrastive and composition sets |
| goff synthetic decisions | 10,304 | 161 hand-written situation seeds × 64 generated examples, German and English, 23% carrying prompt-injection or authority-pressure content labelled with the honest answer. Not public. |
Question types across the mix: 11,232 choice, 7,220 noul, 5,074 score.
| Epochs | 6 (2,100 steps; trained for 4, then resumed for 2 more with the cosine schedule rebuilt over the full length) |
| LR / schedule | 2e-4, cosine, 70 warmup steps |
| Batch | 64, no gradient accumulation, bf16, gradient checkpointing |
| Max length | 1024 tokens; questions whose longest candidate prompt exceeded it were dropped, not truncated (0 of 23,526) |
| Seed | 42 |
| Device | 1 × NVIDIA A100-SXM4-80GB |
| Checkpoint selection | best held-out accuracy; the released checkpoint is the final one (step 2,100), which ties the selected checkpoint on accuracy and is better calibrated |
Held-out validation (the mixed split above, not comparable to the benchmark below): accuracy 0.740, soft accuracy 0.661, soft cross entropy 0.855, Brier 0.309, ECE 0.083, score within-one 0.915.
Compute
Trained on GPU VMs from Lyceum. The full recipe — 6 epochs over 23,526 questions, plus the evaluation passes — runs in about five hours on a single A100 80GB there.
Evaluation
Scored on the LocalLLaMA/typed-decisions
test split (400 cases, 2,000 typed decisions, revision ea930645), ECE at 15 bins,
confidence taken as max candidate probability for every backend. Numbers below are from
the merged weights published here.
| Metric | goff Lite | convaiinnovations/laya (typed-decisions run) |
|---|---|---|
| Accuracy | 0.749 | 0.766 |
| Soft accuracy | 0.517 | 0.471 |
| Soft cross entropy | 0.906 | 0.884 |
| Brier | 0.070 | 0.062 |
| ECE (15 bin) | 0.096 | 0.213 |
| KL | 0.139 | 0.117 |
| Total variation | 0.168 | 0.174 |
| Score MAE | 0.262 | 0.242 |
| Score within-one | 0.990 | 0.995 |
Read this honestly: the Laya comparison checkpoint remains the stronger point predictor
on this split, while this model is markedly better calibrated (ECE 0.096 vs 0.213) and
carries more probability mass on the right candidate (soft accuracy 0.517 vs 0.471).
Per question type, accuracy is 0.732 choice, 0.840 noul, 0.694 score.
Two comparisons worth recording. The previous release, trained on typed-decisions alone, scored 0.720 accuracy / 0.092 Brier / 0.099 ECE on this split; adding the kev and synthetic data and the training changes above moved that to 0.749 / 0.070 / 0.096. And the merged weights reproduce the adapter they came from to within one decision out of 2,000 (adapter: accuracy 0.7495, Brier 0.0700, soft CE 0.9064).
Latency on the benchmark host (A100, batch 128, merged weights): 70.6 ms per case, 14.1 ms per decision.
Serving
vllm serve whadupapp/goff-lite --max-model-len 32768
Score each candidate prompt separately, apply the fitted calibration temperature, then softmax the scores within each question. The temperature must be fitted on a held-out calibration split — not on the training or checkpoint-selection split.
Limitations
- Trained and evaluated only on typed decisions (
choice,noul,score); behavior outside that prompt format is unmeasured. - Evaluated on typed-decisions only. Roughly 44% of the training data is synthetic material generated for goff's own domains, and no public benchmark covers it.
- The window is 32,768 tokens, but no training prompt exceeded 1024; behavior on longer inputs is unmeasured.
- Calibration is fitted downstream; raw scores out of this model are uncalibrated. ECE on the benchmark is 0.096, worse than the 0.071 of an earlier attention-only run — if calibration matters more than accuracy for your use, that trade is worth measuring.
- German coverage is partial: the synthetic portion is mixed German and English, and the benchmark is English-only, so German behavior is unmeasured.
- Latency figures were measured on an A100 and do not transfer to CPU serving.
Provenance
export_manifest.json in this repo records the base revision, adapter SHA-256, exporter
commit, library versions, the fold precision, and the base-vs-merged logit parity check.
The adapter is folded in float32 and the result cast to bfloat16. Parity is measured against the bf16 adapter path, so it includes the cast: max absolute logit difference 0.125 over 8 prompts. That is two bfloat16 representable steps at this model's logit magnitudes (spacing is 0.0625 near 14), and the fold itself is exact — in float32 the merged model matches the adapter to 1.8e-5. The tolerance was set to 0.2 for this export for that reason; a broken fold shows up far above it.