System One Mini
System One Mini is an independent research prototype for five fixed, typed decisions over synthetic controlled software-diagnosis summaries. It is a 69.3M-parameter DistilBERT encoder with five depth-2 classification heads.
This is not a reproduction of Jev or RLCD. It is not a chat model, does not generate text, and cannot answer arbitrary questions.
Intended Use
Use this model to study fixed-schema multi-task classification on complete summaries generated under the documented controlled-intervention policy. It supports only:
| Decision | Options |
|---|---|
task_complete | false, true |
should_retry | false, true |
failure_source | code, environment, test, dependency, unknown |
next_action | inspect_code, rerun_tests, inspect_environment, revert_change, escalate |
evidence_sufficient | false, true |
Do not use it for arbitrary logs, general coding assistance, natural production incidents, causal claims, safety-critical decisions, or autonomous agent control.
Loading
This repository contains reviewed custom Transformers code. Pin a repository revision in production because trust_remote_code=True executes that code.
from transformers import AutoModel, AutoTokenizer
repo = "DavidHatley/system-one-mini"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
state = "Latest verification: checks failed. Acceptance: not confirmed. ..."
tokens = tokenizer(state, truncation=True, max_length=model.config.max_length, return_tensors="pt")
outputs = model(**tokens)
print([logits.softmax(-1) for logits in outputs.logits])
For named, temperature-scaled outputs, download inference.py from this repository and use its load_model and predict functions.
Architecture And Training
- Base:
distilbert/distilbert-base-uncasedat revision12040accade4e8a0f71eabdb258fecc2e7e948be - Shared encoder followed by five independent
Linear -> GELU -> Dropout -> Linearheads - Maximum input length: 256 tokens
- Full fine-tuning for three epochs in BF16 on one NVIDIA RTX 3060
- 20,000 deterministic synthetic training states; seed 1729
- No reinforcement learning, external teacher model, paid API, or real user traces
- Checkpoint selected by validation mean NLL
Evaluation
The post-selection final set uses one deterministic synthetic renderer written and frozen after candidate selection. It was evaluated once. This tests wording transfer inside the same synthetic policy, not real-world diagnosis.
| Final variant | Accuracy | NLL | Brier | ECE | Mean confidence |
|---|---|---|---|---|---|
| Raw | 86.25% | 0.8479 | 0.2630 | 0.1273 | 98.89% |
| Temperature scaled | 86.25% | 0.7133 | 0.3940 | 0.2673 | 63.19% |
Raw per-head results on the final renderer:
| Head | Accuracy | NLL | ECE | Mean confidence |
|---|---|---|---|---|
task_complete | 100.00% | 0.0004 | 0.0004 | 99.96% |
should_retry | 82.65% | 1.1987 | 0.1677 | 99.24% |
failure_source | 92.95% | 0.2982 | 0.0451 | 97.46% |
next_action | 62.80% | 2.4050 | 0.3641 | 99.17% |
evidence_sufficient | 92.85% | 0.3373 | 0.0593 | 98.63% |
The earlier renderer-generalization split was used to compare model candidates and is therefore a development result, not an unbiased final test:
| Development variant | Accuracy | NLL | Brier | ECE | Mean confidence |
|---|---|---|---|---|---|
| Raw | 59.27% | 2.4363 | 0.7666 | 0.3740 | 96.58% |
| Temperature scaled | 59.27% | 0.7875 | 0.4331 | 0.1476 | 62.39% |
Calibration
Per-head scalar temperatures were fitted only on the 2,000-example calibration split. On the final renderer, scaling improved NLL from 0.8479 to 0.7133, but worsened ECE from 0.1273 to 0.2673. Maximum softmax probability must not be interpreted as proven certainty.
Limitations
- Training and evaluation are entirely synthetic.
- The model learns a fixed schema and exact option sets, not arbitrary typed questions.
- Results are from one training seed and one post-selection final renderer.
- Performance is sensitive to wording and representation format.
- Raw predictions are strongly overconfident under distribution shift.
- Temperature scaling did not consistently improve every calibration metric across renderers.
- Inputs longer than 256 tokens are truncated.
Provenance
The model is derived from Apache-2.0-licensed DistilBERT. All added training examples are deterministic project-authored synthetic data. Checkpoint, dataset, calibration, and evaluation hashes are included under provenance/.