this-that-model-1.2

A typed decision model. One forward pass, no decoding loop, no parser, no retry.

1.1 could apply a policy rule. 1.2 can apply it however the rule happens to be written — which is the difference between a model that works on a benchmark and a model that works on the policy document somebody actually wrote.

What changed from 1.1

Composed decisions: where several rules narrow a choice in sequence, accuracy goes from 0.775 to 0.878 on the public benchmark and from 0.785 to 0.909 on the depth ladder.

Rules written in the negative: 1.1 read a rule such as bays without chilled handling are ineligible as though it named the eligible set — not failing to apply the rule but applying its opposite, at 0.88 mean confidence. 1.2 reads it correctly. The table below measures that across eleven phrasings of one rule, because a rule that changes meaning with its wording is the failure a user meets first.

Reading the rule, however it is written

The same 100 decisions, the same gold answers, eleven ways of writing the same eligibility rule. A model that reads rules should score the same on all eleven.

Chance is 0.19. 1.1's range is 0.00 to 0.89; 1.2's is 0.98 to 1.00. Measured on states rendered as indented key/value text, over decisions that filter by eligibility and then rank.

Read the 1.1 column carefully: four of those rows are below chance. A model that scores 0.00 where chance is 0.19 is not guessing — it is reliably choosing the option the rule excludes. On a yes/no question (Is bay-b eligible?) under such a rule, 1.1 answered correctly 6% of the time against a chance rate of 50%.

Where 1.2 sits against models that generate tokens

Every system on all 1,710 questions of the public benchmark, scored the same way: one question, declared options, the answer read back, a system that returns nothing counted wrong.

The benchmark is public: limberc/this-that-complex-decisions. Please noticed that the performance is tested under NVIDIA RTX 5080 GPU (Laptop).

That table measures decision difficulty, and nothing else. On the phrasing table above, claude-opus-5 and gpt-5.6 both score 1.00 on every row, including the rows where 1.1 scores 0.00 — so the frontier models are not weaker at reading rules, they are weaker at these decisions. Two different axes, and this page reports both.

Composed decisions, by depth

Rules that compose — a filter, then a second filter, then a ranking. 242 questions, held out at both the question and the world-state level.

Depth 2 is the hardest rung for the hosted models and the widest gap: two narrowings with many candidates still live is where a single pass over the state pays off most.

The frozen third-party cohort

68 questions a third party recorded against a hosted commercial decision service, in their wording rather than ours:

1.2 misses one of the 68. 1.1 missed none; the cohort is small enough that one question is the whole difference between those two rows, and neither is evidence of much.

Calibration

Where the true answer is a computed probability, so the target follows from the transition rules rather than from anyone's opinion. Lower qL2 is better.

1.1 scored 0.0253 and 0.0203. The second improves; the first is a small regression, and a constant predictor is still three times worse than either.

Usage

pip install git+https://github.com/FLock-io/this-that-model

from thisthat import TypedDecider, Question

decider = TypedDecider.from_pretrained("flock-io/this-that-model-1.2")

answer = decider.decide(
    '{"bays": {"bay-a": {"free": 7, "distance_m": 40, "handling": "chilled"},'
    '          "bay-b": {"free": 3, "distance_m": 12, "handling": "ambient"}},'
    ' "policy": ["Bays without chilled handling are ineligible.",'
    '            "Among those eligible, take the most free positions."]}',
    Question("Which bay is assigned?", ["bay-a", "bay-b", "hold in transit"]),
)

print(answer.choice)          # bay-a
print(answer.confidence)      # threshold on it; see the calibration section

That is the example 1.1 returned bay-b for, at 0.9998 confidence, and it is the one this release exists to fix. Over 40 freshly generated decisions of exactly that shape — JSON, two candidates, the rule written as an exclusion — 1.1 answers 0.20 and 1.2 answers 0.975. Stated positively, both answer 1.000.

Citation

@misc{cheng2026thisthat,
  title  = {A typed decision model that decides in 50 ms, for a millionth of a cent},
  author = {Cheng, Zehua and Dai, Wei and Sun, Jiahao},
  year   = {2026}
}

Licence

MIT. Adapted from decider-2b under Apache-2.0.