Schema-conditioned candidate scorer (DeBERTa-v3-large)
One DeBERTa-v3-large encoder with a single scalar head scores (state, question + candidate) pairs.
Deterministic code groups the scalar logits per question and decodes them into three answer primitives:
| primitive | input schema | answer |
|---|---|---|
choice | criteria: {option_id: description} | argmax option id + probabilities |
noul | optional criteria: {"true": ..., "false": ...} | p(proposition is true) |
score | criteria: [level 0 description, level 1, ...] | expected level index + per-level probabilities |
The question text, criteria and option ids are read at inference time, never baked into the weights, so the same checkpoint answers new questions over new label sets without retraining.
Try it in the Space: mobarmg/jev-schema-scorer.
How it works
For every candidate of a question the model sees a sentence pair:
sequence_a = the state (free text, or a JSON object serialised)
sequence_b = {"candidate": {"id": "<option id>", "description": "<option description>"},
"type": "choice", "instructions": "...", "criteria": {...}}
The candidate sits right after [SEP], so the only tokens that differ between a question's candidates
are where the encoder attends most easily. Each pair yields one logit; a softmax over the question's
candidates gives the answer distribution. Training minimises cross-entropy between that grouped softmax
and a target distribution (one-hot for choice / score, [1-p, p] for noul).
Usage
The repo ships schema_scorer.py with the request compiler, decoder and a small adapter.
from huggingface_hub import hf_hub_download
import importlib.util
repo = "mobarmg/jev-schema-scorer-deberta-v3-large"
spec = importlib.util.spec_from_file_location("schema_scorer", hf_hub_download(repo, "schema_scorer.py"))
schema_scorer = importlib.util.module_from_spec(spec); spec.loader.exec_module(schema_scorer)
scorer = schema_scorer.LocalSystemOne(repo)
scorer.system_one(
"Nine days of silence on a signed quote is a joke. Our launch event is on the 28th and we still "
"do not have the licence keys your sales team promised. Somebody pick up a phone.",
{
"department": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "Payment or subscription issues",
"technical": "Bugs or integration problems",
"sales": "Pricing or plan questions"}},
"frustration": {"type": "score", "instructions": "How frustrated the customer appears",
"criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"]},
"is_urgent": {"type": "noul", "instructions": "The message conveys urgency"},
},
)
# {'answers': {'department': {'type': 'choice', 'choice': 'sales', 'probabilities': {...}},
# 'frustration': {'type': 'score', 'score': 2.0, 'probabilities': [...], 'legend': {...}},
# 'is_urgent': {'type': 'noul', 'noul': 0.99}}}
Without the helper, it is a plain DebertaV2ForSequenceClassification with num_labels=1: tokenize
(state, serialised question + candidate) pairs, take logits[:, 0], and softmax over each question's
candidates.
Limits: state + schema + candidate must fit in 512 tokens (trained at 384); every question needs at least two candidates.
Training data
Fine-tuned from microsoft/deberta-v3-large (supervised, grouped-softmax cross-entropy, max 384 tokens)
on the union of two synthetic English datasets, 17,170 training questions in total:
- v1 (7,650 train / 1,350 eval questions, 3,000 texts, 30 domains): support triage, moderation,
product reviews, email routing, resume screening, banking, insurance claims, telehealth messages,
IT helpdesk, dating safety, and more. One
choice, onescoreand onenoulquestion per domain. - v2 (9,520 train / 1,680 eval questions, 2,400 states, 24 domains): harder task shapes. JSON-object
states (orders, transactions, API request logs, candidate profiles), multi-turn dialogue transcripts
(support chats, sales calls, tutoring), code snippets and log excerpts,
choicequestions with 5-7 options including an "other" bucket, 4-5 levelscorequestions,noullabels defined by explicit rules over the state, sarcasm and negation, summary faithfulness, and rubric grading of student answers. Each domain has 4-6 questions.
To make the model read the schema instead of memorising label ids, each training example draws a
random instruction wording and criteria wording, shuffles the choice options, and replaces the option
ids with opaque ids (opt_a, k2, bravo, ...) half of the time. Eval splits hold out 15% of records
per domain.
Evaluation
Measured with bench_dataset.py on the held-out eval splits.
| eval split | metric | this checkpoint | previous checkpoint (v1 only) |
|---|---|---|---|
| v2 (1,680 questions) | choice accuracy | 0.841 | 0.687 |
| v1 (1,350 questions) | choice / noul / score | unchanged from the previous checkpoint | 0.889 acc / 0.940 acc, 0.052 Brier / 0.183 MAE |
Chance on v2 choice is 0.214; on v1 it is 0.255. The earlier revision of this repo (v1-only training)
remains available in the commit history.
Known weak spots (near chance): order_record.next_action, summary_faithfulness.error_type, and
student_answer grading.
Intended use and caveats
- Intended for experiments with schema-conditioned classification: routing, triage, moderation-style labelling, ordinal scoring, and yes/no propositions over short English texts.
- Trained on synthetic data. Expect degraded accuracy far from the training domains, on long inputs, on non-English text, and on criteria that require world knowledge or reasoning across the text.
- Probabilities are often very peaked (the grouped softmax is trained on one-hot targets); treat them as rankings rather than calibrated confidences.
- Not a safety classifier. Do not use its
moderation,health, orsecurityoutputs to make consequential decisions without human review.