gemma3-270m-typed-decisions
Gemma 3 270M fully fine-tuned as a probabilistic decision model for
LocalLLaMA/typed-decisions. Mode: fine-tuned on train (all four workflows,
1080 cases for training and 120 held out for epoch selection on validation KL).
Results
Scored on LocalLLaMA/typed-decisions all/test (400 cases, 2000 decisions), never seen in training.
| Model | Kind | Accuracy ↑ | KL from gold ↓ | Brier ↓ | ECE ↓ | Latency |
|---|---|---|---|---|---|---|
| gemma3-270m-typed-decisions (Gemma 3 270M, full fine-tune) | fine-tuned on train | 0.761 | 0.1319 | 0.0716 | 0.1480 | 68 ms / case (cuda) |
ECE is computed per question and averaged over the 20 questions (closest to the README's reference rows); pooled over all decisions it is 0.1146.
By question type
| accuracy | kl | brier | ece | |
|---|---|---|---|---|
| choice | 0.732 | 0.167 | 0.085 | 0.112 |
| noul | 0.822 | 0.089 | 0.062 | 0.066 |
| score | 0.739 | 0.138 | 0.068 | 0.158 |
| OVERALL | 0.761 | 0.132 | 0.072 | 0.115 |
By workflow
| accuracy | kl | brier | ece | |
|---|---|---|---|---|
| agent_trace_observability | 0.740 | 0.095 | 0.057 | 0.163 |
| customer_service | 0.772 | 0.134 | 0.070 | 0.114 |
| invoice_processing | 0.784 | 0.217 | 0.115 | 0.087 |
| security_incidents | 0.750 | 0.081 | 0.045 | 0.151 |
| OVERALL | 0.761 | 0.132 | 0.072 | 0.115 |
Baselines on the same split: Prior accuracy 0.478 / KL 0.327; zero-shot Gemma 3 270M accuracy 0.279 / KL 1.068.
Full test distributions: test_predictions.jsonl; metrics: test_metrics.json.
Gold is the mean of three teacher samples, so scores measure agreement with that teacher (self-agreement ≈ 0.735).
How it works
Each (case, question) pair is one prompt: the case JSON, the question's instructions, and its options listed as
letters A, B, C, … with their descriptions. The model's next-token logits at the reply position, restricted to
those letter tokens and softmaxed, are the answer distribution. Training minimises soft cross-entropy against the
gold distribution. One forward pass per question (5 per case).
Usage
import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "rony/gemma3-270m-typed-decisions"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, attn_implementation="eager").eval()
LETTERS = "ABCDEFGHIJ"
SYSTEM = 'You are a precise decision model. Read the case, then answer the question with the single letter of the best option.'
def options(q):
if q["type"] == "choice": return list(q["criteria"].items())
if q["type"] == "noul":
c = q.get("criteria") or {}
return [("true", c.get("true", "The statement is true.")), ("false", c.get("false", "The statement is false."))]
return [(str(i), d) for i, d in enumerate(q["criteria"])]
@torch.no_grad()
def decide(state, questions):
out = {}
for name, q in questions.items():
opts = options(q)
kind = {"choice": "Choose one option.", "noul": "Is the statement true or false?",
"score": "Pick the level on the rubric."}[q["type"]]
prompt = (f"{SYSTEM}\n\n### Case\n{json.dumps(state, separators=(',', ':'))}\n\n"
f"### Question ({name})\n{q['instructions']}\n{kind}\n\n### Options\n"
+ "\n".join(f"{LETTERS[i]}. {l}: {d}" for i, (l, d) in enumerate(opts))
+ "\n\nAnswer with one letter.")
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
logits = model(**tok(text, add_special_tokens=False, return_tensors="pt")).logits[0, -1]
ids = [tok.encode(LETTERS[i], add_special_tokens=False)[0] for i in range(len(opts))]
p = torch.softmax(logits[ids].float(), -1).tolist()
out[name] = dict(zip([l for l, _ in opts], p))
return out