udaykamath/OpenSystemOne-ModernBERT-base-v0.1
An open System One classifier: typed questions about a piece of context, answered with calibrated probabilities. Yes/no (Noul), pick-one (Choice) and ordered-scale (Score) questions, many per call.
Inspired by the public description of TypeSafe AI's Jev. Not affiliated with TypeSafe AI and not a copy of Jev.
s1 = SystemOne.from_pretrained("udaykamath/OpenSystemOne-ModernBERT-base-v0.1")
s1({"message": "I was charged twice and need a refund today."},
{"urgent": Noul("`message` communicates time pressure."),
"intent": Choice("What does the customer want?", {"refund": "money back", "technical_help": "a bug fixed"})})
| Design | cross-encoder: state and question read together |
| Backbone | tasksource/ModernBERT-base-nli |
| Context | 512 tokens of state, 128 of question |
| Temperature | 1.402 (fitted on held-aside training-mix tasks) |
Held-out results (zero-shot)
| task | acc | macro_F1 | ECE |
|---|---|---|---|
| IMDb (noul) | 0.925 | 0.925 | 0.028 |
| IMDb flipped (noul) | 0.921 | 0.921 | 0.019 |
| IMDb (choice) | 0.940 | 0.940 | 0.025 |
| AG News (choice) | 0.870 | 0.869 | 0.076 |
| TweetEval-hate (noul) | 0.503 | 0.457 | 0.250 |
| SST-5 (score) | 0.487 | 0.492 | 0.159 |
| SMS spam, 4 wordings (noul) | 0.818 | 0.763 | 0.030 |
| Banking77, 8 options (choice) | 0.844 | 0.843 | 0.162 |
| Yahoo Answers topics (choice) | 0.674 | 0.667 | 0.114 |
Robustness
- negation: mean |P(liked)+P(disliked)-1|: 0.088
- negation: correlation: -0.929
- paraphrase: mean std of P(yes): 0.030
- paraphrase: worst-wording accuracy: 0.880
- paraphrase: best-wording accuracy: 0.920
- option order: same answer under 4 rotations: 0.963
- none trap: picks 'none' when the answer is missing: 0.853
- none decoy: accuracy with 'none' added: 0.777
- labels: accuracy with bare names: 0.873
- labels: accuracy with descriptions: 0.860
- context: IMDb accuracy at 128 tokens: 0.860
- context: IMDb accuracy at 256 tokens: 0.897
- context: IMDb accuracy at 512 tokens: 0.913
Training data and licences
see repository
Some training sources carry non-commercial terms. See DATA_LICENSES.md in the code repository before commercial use.
Limitations
- Each question is a separate pass over the state, so 16 questions cost about 16 times one.
- Offensive and hateful text are not well separated: on TweetEval-hate the model ranks hate reasonably (AUC about 0.7) but says yes far more often than the labels do.
- Literal cues (explicit urgency, dates, numbers) can be missed.
- Calibration was fitted on the training mix and can drift on a new domain. Re-fit the temperature on your own sample.
- English only.