system-one-270m
A System One model: it takes a piece of state plus a typed question with a caller-supplied option set, and returns one option with a calibrated probability. It never generates free text.
Built as an open reproduction of the idea behind TypeSafe's
Jev (closed)
and convaiinnovations/laya
(ModernBERT-large), on a 270M decoder rather than an encoder.
How it answers
Options are rendered as letters in the prompt. After Gemma's
<start_of_turn>model\n a bare letter is exactly one token, so a single forward
pass yields logits over the whole answer space at once. The head gathers the
first n_options letter ids, masks the rest to -inf, and softmaxes over that
slice.
- Type safety — an option outside the caller's set is not representable.
- Calibration — trained against soft targets with the log score, a strictly proper scoring rule, so honest probabilities maximise the objective.
- Flat latency in option count — options are input, not N generations.
import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer
LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
tok = AutoTokenizer.from_pretrained("kaivoss/system-one-270m")
model = AutoModelForCausalLM.from_pretrained("kaivoss/system-one-270m").eval()
def decide(state, question, options):
body = "\n".join(f"{LETTERS[i]}. {o['label']} - {o['description']}"
for i, o in enumerate(options))
prompt = (f"<state>\n{state}\n</state>\n\nQuestion: {question}\n"
f"Options:\n{body}\n\nAnswer with one letter.\nAnswer:")
text = tok.apply_chat_template([{"role": "user", "content": prompt}],
tokenize=False, add_generation_prompt=True)
ids = torch.tensor([tok.encode(text, add_special_tokens=False)])
with torch.no_grad():
logits = model(input_ids=ids, logits_to_keep=1).logits[:, -1, :]
letter_ids = torch.tensor([tok.encode(LETTERS[i], add_special_tokens=False)[0]
for i in range(len(options))])
probs = F.softmax(logits.index_select(1, letter_ids), dim=-1)[0]
best = int(probs.argmax())
return options[best]["label"], float(probs[best])
print(decide(
"My card was charged twice for the same month and support hasn't replied.",
"Which team should handle this?",
[{"label": "billing", "description": "Payment or subscription issues"},
{"label": "technical", "description": "Bugs or integration problems"},
{"label": "sales", "description": "Pricing or account questions"}]))
Results
2,493 held-out questions, split by state so no state appears in both train and eval.
| Metric | Base (gemma-3-270m-it) | This model |
|---|---|---|
| Accuracy | 0.4204 | 0.6574 |
| Brier | 0.7182 | 0.4110 |
| ECE | 0.3035 | 0.1311 |
| ECE (temperature-scaled, T=2.0) | 0.1062 | 0.0374 |
| Random baseline | 0.3471 | 0.3471 |
Calibration matters more than accuracy for this model class. A model right 66% of the time that says 66% is usable; one that says 99% is not. The ECE drop is what the proper-scoring-rule objective bought.
Training data
25,002 synthetic typed decisions over 7,537 states, generated with
openai/gpt-oss-20b via OpenRouter at ~$0.000275/question. 28 domains;
49.3% choice, 33.4% noul (yes/no), 17.3% score (ordinal).
Soft targets come from an ensemble of permuted reads: gpt-oss reasons before answering and cannot be told not to, which makes any single response's logprobs effectively one-hot. The uncertainty lives in the variance across reads, and shuffling option order each time also cancels letter-position bias.
Training: 3 epochs, bs 32, lr 5e-5 cosine, ~24 min on one H100 SXM.
Limitations
- 84.6% of training rows are unambiguous, so graded uncertainty is under-represented and calibration on genuinely contested inputs is the weakest part of the model.
- Accuracy is below Laya's reported 0.766 (on its own benchmark, which is not this eval set — the two numbers are not directly comparable).
- Option sets above ~14 are barely represented (2.6% of rows at 9–14, 0.04% above 15); the letter scheme caps at 26.
- Labels are model-generated, not human, so the ceiling is gpt-oss-20b's own judgement, and any systematic bias it has is baked in.
- State is not treated as hostile. Text inside the state can steer the answer; this is an injection surface, exactly as TypeSafe documents for Jev.
- English only in practice, though nothing in the design is language-specific.