System One (zeroshot)
A 70.8M-parameter DeBERTa-v3-xsmall cross-encoder that answers typed decision questions, Choice,
Score and Noul, in one batched forward pass with zero generated tokens. It implements the
TypeSafe Jev POST /v1/systemone schema and is meant for the routing, classification and gating
decisions inside software that do not need an autoregressive model.
Supervision for this checkpoint: None. The base NLI checkpoint, served as is.
Accuracy on the 500-item test splits
| question | accuracy | majority floor |
|---|---|---|
intent | 0.508 | 0.010 |
tool | 0.384 | 0.306 |
needs_tool | 0.792 | 0.694 |
priority | 0.464 | 0.464 |
type | 0.560 | 0.374 |
is_incident | 0.754 | 0.626 |
Read these next to the floor column, not on their own. Two of the six questions are at or below a constant predictor for every system measured, Claude Haiku 4.5 included, and that is a finding about the labels rather than about any model. The repository documents a retraction of one of our own earlier headline numbers for the same reason.
Use
from systemone.engine import SystemOneEngine
from systemone.schema import SystemOneRequest
engine = SystemOneEngine('shreyanbr/system-one-zeroshot')
request = SystemOneRequest.model_validate({
'state': {'message': 'my card still has not arrived'},
'questions': {'intent': {'type': 'choice', 'instructions': 'What does the customer want?',
'criteria': {'card_arrival': 'a card that has not arrived',
'refund': 'a refund request'}}},
})
print(engine.answer(request).answers['intent'])
calibration.json ships alongside the weights and is required: the raw margins are not
probabilities, and the engine reads a per-question temperature and a Platt scaler from it.
Caveats worth reading before you trust a number
- The base checkpoint was trained on banking77, so that row is contaminated. banking77 also has a published label-error rate near 14%, capping achievable accuracy around 0.86.
triage.priorityis close to unlearnable on this data. Every backend sits at or below the floor.- The
ticketsdataset is CC-BY-NC-4.0, so thetriagesupervision is not for commercial use. - Single seed, single run. Intervals are bootstrap over test items, not over training seeds.
Source, benchmark harness and the full limitations section: https://github.com/shreyanbr/jev-haiku-benchmarking