Dinah-0

A 150M-parameter encoder that makes typed decisions, in English and Portuguese, on a CPU.

Answers the same typed-decision format as Jev (choice, noul, score, with probabilities and confidence). Not affiliated with TypeSafe.

Need commercial use? Dinah-0 Apache is the same Dinah-0 retrained without the non-commercial data and released under Apache-2.0. It scores about 0.2 points lower on the Decision Index 0.3 (25.86 against 26.06).

• Best model under 1B parameters on the Decision Index 0.2.1 (27.63; the next one under 1B scores 19.22)
• Best encoder on the board (Decision Index 0.2.1)
• Ahead of models with 2B to 4.7B parameters (Decision Index 0.2.1), running on a CPU with ~150 MB on disk
• On the current Decision Index 0.3 (live board, 2026-10-10): public index 26.06, full score 13.87. The three points above are for edition 0.2.1; the 0.3 board has more entrants, including a 0.9B model above Dinah-0 on the public index.

The full story (how it was trained, evaluated and what went wrong along the way): lukita.me/posts/dinah-0-en (in Portuguese: lukita.me/posts/dinah-0).

Dinah-0 reads a question (instructions, state and a set of options) and returns a probability distribution over the options. The same model answers three question types:

It does not generate text. It is built for classification, routing, tool selection, extraction checks and similar decisions where a small, fast and cheap model is enough.

Results

Decision Index 0.2.1

Measured with the official kit and scorer (commit 87d4650) and the engine in this repository (decision_index_engine.py at revision e6bfc33), on the full suite (38 benchmarks, 150,759 requests), in bfloat16 on one RTX PRO 4000 Blackwell. Nothing was truncated: the 324 requests longer than 8,192 tokens were refused and count as wrong. Self-reported; the submission to the official leaderboard is pending. Median latency in that run: 5.6 ms per request.

On the live leaderboard of 2026-09-28 this would rank 41st of 72, the best model under 1B parameters (the next one scores 19.22) and the best encoder, ahead of several models between 1.7B and 4.7B parameters.

Note on GSM8K: about 1.9 of these points come from a shortcut in the benchmark's multiple-choice GSM8K (the gold answer is always near the middle of the options), which Dinah-0 learned because its training used options in the same style. Without it the index would be 25.72 (43rd of 72), still the best under 1B and the best encoder. Reported to the maintainers in decision-index#32.

Other evaluations

Dinah-0 was trained on the typed-decisions train split and on Lichess puzzles (with the test sets held out), so these are in-distribution numbers. The typed-decisions labels come from a teacher model, so accuracy there measures agreement with that teacher.

Speed

Measured on an idle machine with an earlier checkpoint of the same architecture (latency depends only on the architecture). Peak memory is ~750 MB at 512 tokens and over 6 GB at 8,192.

Usage

dinah.py (in this repository) has the full input format and two backends with the same API.

pip install torch transformers safetensors huggingface_hub   # PyTorch backend
pip install onnxruntime transformers huggingface_hub         # ONNX int8 backend (CPU)

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("Lukitaduarte/dinah-0", "dinah.py")
spec = importlib.util.spec_from_file_location("dinah", path)
dinah = importlib.util.module_from_spec(spec); sys.modules["dinah"] = dinah; spec.loader.exec_module(dinah)

model = dinah.DinahONNX.from_pretrained("Lukitaduarte/dinah-0")   # or dinah.Dinah for PyTorch

model.choice(
    state="The package arrived broken and I need it for tomorrow.",
    instructions="What does the customer want?",
    criteria={
        "refund": "The customer wants their money back",
        "replacement": "The customer wants a new unit sent",
        "complaint": "The customer only wants to complain",
    },
)
# {"type": "choice", "choice": ..., "confidence": ..., "probabilities": {...}}

model.noul(
    state="Hi, please cancel my subscription starting today.",
    instructions="Does the customer want to cancel?",
    true="The customer wants to cancel.",
    false="The customer does not want to cancel.",
)
# {"type": "noul", "noul": ...}
model.score(state="Great product, but shipping took three weeks.", instructions="Rate the review.",
            criteria=["very negative", "negative", "neutral", "positive", "very positive"])

state, instructions and each option can be a string or a JSON object/array. For noul, describe both poles (true and false): with only the statement, the model has a generic "the statement above is false" to compare against and is much less reliable. Several questions can be answered in one call with model.predict([...]).

Input format

Each question becomes one sequence, and each option is scored from the hidden state at its [OPT] marker:

[CLS] instructions [SEP] state [SEP] [OPT] option_1 [OPT] option_2 ... [OPT] option_n

Objects are serialized as compact JSON with sorted keys. For noul, the options are [false, true] and the answer is the probability of true.

Files

The int8 model agrees with fp32 on the top option in 94.3% of 300 typed-decisions test items, with the same accuracy (73.7% vs 73.0% on that sample). Use the PyTorch weights when you need the exact fp32 probabilities.

The int8 model's accuracy depends on the ONNX Runtime version: on ORT 1.23 (the version mobile plugins ship) it agrees with fp32 on only 75.7% of the same kind of sample (69.0% accuracy). onnx/model_8bit.onnx agrees on 97.0% there (74.7% accuracy), at 149 ms per request on an Apple M5 CPU against 46 ms for int8. Prefer it on mobile.

Model details

• Architecture: ModernBERT-base (22 layers, hidden size 768, 8,192-token context), ~150M parameters, with an [OPT] token added to the vocabulary, a question-type embedding, an option scorer (MLP over each [OPT] hidden state) and a small confidence head that reads the shape of the option distribution.
• Base model: Tropic-AI/moBERTo (Apache-2.0), a ModernBERT further pre-trained on Portuguese, in the variant that keeps the original ModernBERT tokenizer.
• Training: several fine-tuning stages on typed-decision data (English and Portuguese), long-context multitask data, Lichess puzzles and, in the last stage, 605,256 examples built from the training splits of the Decision Index source datasets (never from the suite's evaluation rows), with Portuguese replay. About 1.19 billion training tokens across all stages.
• Objective: cross-entropy against the target distribution over options, plus a Brier term; the confidence head is trained to predict whether the top option is correct.

Limitations

• Decisions only. No generation, no open-ended answers.
• Little world knowledge. It is weak on questions that need facts it cannot read in the input (see the Knowledge and Arts scores above).
• Weak at arithmetic over the input. Comparing numbers in the state (totals against limits, dates) is unreliable.
• Two languages. Trained for English and Portuguese; other languages are untested.
• 8,192 tokens, no truncation. Longer requests raise an error instead of being silently cut.
• Confidence is not recalibrated in this release. The confidence value comes straight from the confidence head.
• Research model. Evaluate it on your own data before relying on it.

License

CC BY-NC 4.0 (non-commercial). Part of the last training stage comes from the MMLU auxiliary training set, which includes RACE, released for non-commercial research only, and MCTest. Because of that, these weights are released for non-commercial use.

Training data and attributions

Datasets in the training path, with their licenses:

Thanks to the authors of all of them, and to the ModernBERT and moBERTo teams for the base model.

Author

Built by Lukita (GitHub) as a personal study project.