gliner-datause-v3-probe
Multi-origin probe of rafmacalaba/gliner_datause (proposals: NAMED_DATA,
DESCRIPTIVE_DATA, VAGUE_DATA). The Luna v2.4 keep/drop boundary
(v2.4-patched-2026-09-05) is read off the frozen representation with a small MLP head
on [start; end; mean; ±64-token window] span features:
extractor proposes, head disposes.
Trained on gold v2 (2,728 Luna-verdict spans over
rafmacalaba/datause-extracted-sample, gliner2 config: FCV PADs, JDC
operational, refugee PADs, ReliefWeb, JAD PADDY, PRWP) plus 26,618
campaign-v2 Luna single-judge labels drawn origin x specificity x
0.1-decile across the full extraction corpus — 29,346 spans total,
doc-disjoint 70/15/15 split, every origin in train, val, and holdout, no
passage crosses splits. The full candidate pool (2,729 sample spans +
27,126 campaign spans, passages included) is versioned alongside the
splits in rafmacalaba/datause-probe-v3 (candidates config). BCE
targets are label-smoothed by 0.1 (eps/2) to curb
overconfident scores; scores are ranking-oriented, not calibrated
probabilities.
Results (holdout, gold = Luna distillation verdicts)
- head AUROC: 0.8867 (distillation fidelity: how well the head carries the Luna v2.4/v2.5 keep/drop boundary)
- head best-F1 thr 0.3: precision 0.7481 · recall 0.9312 · f1 0.8297
- val AUROC: 0.8915
Per origin
| origin | n | keep | AUROC | head best-F1 @ thr |
|---|---|---|---|---|
fcv_pads_east_africa | 1123 | 431 | 0.9110 | 0.7961 @ 0.5 |
general_prwp | 1859 | 1299 | 0.8611 | 0.8721 @ 0.3 |
jad_paddy_docs | 169 | 158 | 0.9574 | 0.9783 @ 0.3 |
jdc_operational | 242 | 115 | 0.9068 | 0.8128 @ 0.5 |
refugee_pads | 460 | 210 | 0.9231 | 0.8317 @ 0.6 |
reliefweb | 804 | 447 | 0.7665 | 0.7586 @ 0.3 |
Human check (never trained on)
| set | n | keep | AUROC | head best-F1 @ thr |
|---|---|---|---|---|
human473 | 473 | 315 | 0.8508 | 0.8477 @ 0.3 |
annotator190 | 190 | 110 | 0.9285 | 0.8772 @ 0.4 |
jdc283 | 283 | 205 | 0.7980 | 0.8620 @ 0.1 |
Labels: gold-v2 verdicts + campaign-v2 Luna single-judge labels
(v2.4-patched-2026-09-05, keyed single-span batch, prompt v2.5-draft-2026-09-06);
they are a distillation target, not human gold. 7 spans quarantined
(unresolved echo) + 1 bad_json excluded — see quarantine.jsonl in
rafmacalaba/datause-probe-v3. Head: head.pt; per-span holdout
predictions: holdout_predictions.jsonl (key, origin, label,
probe_score, head_score); full sweeps: holdout_metrics.json. Human
external check: human473_predictions.jsonl (473 annotator190/jdc283
spans, never trained on) with per-set AUROC + best-F1 in
holdout_metrics.json → human473 and the table above.