kambo-v1-decide

kambo-v1-decide is Kambo-v1 fine-tuned to make typed decisions: you give it a state (a message, a document, a log line) and a set of questions about it, and it returns a probability for every allowed answer. It doesn't generate text.

Each question has a type:

Every answer is read off one forward pass. The state is encoded once and shared by all questions in the call. Each option gets a single-token label, so all options are scored from one logit vector.

Usage

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("VikramPal/kambo-v1-decide")
sys.path.insert(0, path)
from kambo_decide import Decider, Choice, Noul, Score

d = Decider(path)
out = d.decide(
    "Customer: I was charged twice for my card this month and nobody answers. "
    "Fix it or I'm closing my account.",
    [
        Choice("Which team should handle this?", ["billing", "technical support", "sales"]),
        Noul("Does the message mention a payment card?"),
        Score("How angry is the customer?", ["calm", "annoyed", "furious"]),
    ],
)
for r in out:
    print(r["question"], "->", r["answer"], r["probs"])

Each result also carries confidence (top probability) and label_mass: the share of the raw next-token distribution that landed on a valid label. A low label_mass means the input didn't fit the question well, and it's a usable out-of-distribution signal on its own.

The Decider scores each question in the given option order and in reversed order, then averages the two. This cancels the position bias that otherwise favours option "A". To skip that and halve the cost, pass permute=False.

Tested with transformers==4.53.2 and torch 2.x. On a single A100, four questions about one state take about 150 ms in one call. A 77-option question takes about the same.

Adding a "none" option

The model was trained with a "none of these" option that is sometimes the correct answer. Add one to any Choice when the input may not fit any of your options:

labels = ["card arrival", "lost or stolen card", "exchange rate", "none of these"]
d.decide("Customer message: who won the 1998 world cup?",
         [Choice("Which banking request is the customer making?", labels)])

Optional calibration

decide(..., calib={"choice": {"T": t, "b": offsets}}) applies a temperature and per-option offsets fitted on your own labelled data. kambo_decide.fit_temperature and kambo_decide.fit_bias fit them from the output of Decider.raw_logprobs. The model is usable without this: the raw probabilities in the tables below are uncalibrated.

Results

All numbers come from 400 test items per task, run on one A100 in bf16. "Calibrated" means a temperature plus per-option offsets fitted on a separate 200-item dev split of the same task. The baseline is the unmodified Kambo-v1, scored with the same prompts and code.

Out-of-domain inputs: we asked the 77-way Banking77 question about 200 trivia questions (SQuAD validation) and about 200 real banking messages.

Without a "none" option, the model doesn't force a confident answer onto off-topic input. Its confidence drops to 0.11 on average, so a confidence threshold separates in-domain from off-topic inputs.

Training

This is a full fine-tune of every weight except the MoE routers, which are frozen so the expert balance learned in pretraining is kept. The architecture is unchanged.

• Objective: cross-entropy over the option labels only, plus 0.1 × cross-entropy of the correct label over the full vocabulary. The first term is the expected log-score of the answer distribution, so it rewards calibrated probabilities rather than just a correct top pick. The second keeps probability mass on valid labels.
• Schedule: 2,500 steps, batch size 32 (80,000 examples), AdamW (β = 0.9, 0.95), learning rate 1e-5 with 100 warmup steps and cosine decay, gradient clip 1.0, bf16 autocast with fp32 master weights. About 2.1 hours on one A100 40 GB.
• Data: up to 8,000 examples from each source, train splits only. The examples are recast as Choice, Noul or Score questions. Options are randomly subsampled and shuffled. With probability 0.15 the true answer is removed and "none of these" becomes correct, and "none of these" is often added as a distractor.

Sentiment data was deliberately left out, so IMDB measures transfer to an unseen task. The training datasets keep their own licenses. Notably, ANLI is CC BY-NC 4.0.

Limitations

• English only. The questions and states in training were all English.
• It decides from what is stated. It is weaker on implicit cues: for "Fix it or I'm closing my account", P(threatening to leave) is 0.46.
• Each task was measured on 400 items with one training seed, so differences of a couple of points are within noise.
• Calibrating on a 200-item dev set doesn't always help: on BoolQ it raised accuracy but worsened ECE (0.054 raw → 0.086). Calibrate on more data, or use the raw probabilities.
• Don't use it as the only signal for high-stakes decisions.

License

Apache-2.0, the same as the base model.