Sieve-0.8B
Sieve-0.8B is a decision model. It takes a state, a typed question and a closed set of options, and returns a calibrated probability for every option. It generates no text. It is the small member of the Sieve family (Sieve-2B-v2, Sieve-4B, Sieve-9B-v2, Sieve-27B): the same architecture, here fully fine-tuned rather than trained with LoRA.
| parameters | |
|---|---|
| run at inference (embeddings + blocks 1-16 of the base, scorer head) | 0.59 B |
| trained (blocks 1-16 + scorer head; embeddings frozen) | 332.6 M |
full base model (Qwen/Qwen3.5-0.8B-Base text model), for reference | 0.75 B |
Not affiliated with TypeSafe AI.
How it works
The state and the question are encoded once. Every option is listed in the prompt, in an order fixed by a hash of each option's text. The encoded state is then copied once per option, and each option is scored as its own continuation of that copy. No option sees another, so reordering the options cannot change the answer: that holds by construction, not by training. A scalar head reads the last token of each option; a softmax over the options, divided by one built-in temperature, gives the probabilities.
from modeling_sieve import Sieve # needs: transformers==5.17, torch, safetensors, huggingface_hub
s = Sieve.from_pretrained("sthanika-ai/Sieve-0.8B") # downloads Qwen3.5-0.8B-Base once, applies the released block weights + head
s.choice(
state="Customer: my invoice was charged twice and nobody answers the phone!",
question="Which team should handle this?",
options=["Billing", "Technical support", "Sales"],
)
Yes/no questions are answered as a 2-option choice. Inputs up to 32,768 tokens are scored; training used inputs up to 8,192.
The repository ships the weights of blocks 1-16 (full_weights.safetensors) and modeling_sieve.py loads them over the base model.
Training
- Base:
Qwen/Qwen3.5-0.8B-Base, a hybrid of Gated DeltaNet and full-attention blocks; only blocks 1-16 of 24 run. - Method: full fine-tune of blocks 1-16 (332.1 M parameters, fp32) plus a scalar head (LayerNorm → 512 → 1); the token embeddings stay frozen. Trained through the same option fork that serves the model.
- Recipe: 1 epoch (5,947 steps), AdamW, lr 5e-6 with 3% warmup and cosine decay to 10%, batch 32 questions, gradient-norm clip 1.0. The learning rate was chosen on a 38,060-question pilot (1e-5, 5e-6 and 2.5e-6 tried; 2e-5 was clearly worse) and compared with a LoRA run of the same size.
- Data: 190,296 questions from 155 decision tracks of a public + synthetic mix, ordered so every part of the epoch sees
every track. It covers intent and topic classification, NLI and fact-checking, commonsense and knowledge questions, safety
and moderation, preference judging, tool selection and routing, retrieval and long documents, math word problems
and rule-based synthetic reasoning (dates, numbers, policies,
lookups), plus synthetic phishing-email decisions written for this mix.
- 32,181 questions (16.9%) come from the train or dev splits of datasets whose test splits appear in the Decision Index: iSarcasmEval 6,803, HellaSwag 5,000, WinoGrande 5,000, CLINC150 3,227, GSM8K 3,167, ESCI 2,989, BANKING77 2,403, RAGTruth 1,378, HoVer 1,308, When2Call 464, ContractNLI 442. No test split was used.
- Each public dataset keeps its own licence.
Calibration
One temperature, T = 1.1208, fitted by NLL on a 10,013-question calibration split that covers every track and was never trained on. It never changes which option wins. On a separate 2,388-question validation split (accuracy 75.8%):
| NLL | ECE | confidence − accuracy | |
|---|---|---|---|
| raw (T = 1) | 0.650 | 0.029 | +0.013 |
| built in (T = 1.12) | 0.645 | 0.024 | −0.005 |
Decision Index
Self-run with the official kit, edition 0.3 public suite (37 benchmarks, 140,178 requests): public index 32.26, complete, 2 requests refused (options with identical text). The 0.3 board's main ranking also includes private tests that only the maintainers run.
| area | 0.3 public | 0.2.1 |
|---|---|---|
| Knowledge & Reasoning | 14.3 | 11.9 |
| Language Understanding | 26.6 | 26.8 |
| Retrieval & Classification | 54.8 | 54.8 |
| Tools & Automation | 51.2 | 51.2 |
| Arts & Human Taste | 13.5 | 11.3 |
| Index | 32.26 | 31.46 |
Results: sthanika-ai/Sieve-0.8B-decision-index-results.
Latency (single process, one request at a time, bf16, A100 80GB, 400 uniformly sampled suite requests): median 69 ms, mean 92 ms, 80th percentile 91 ms.
Limitations
- Knowledge-heavy and multi-step benchmarks (MMLU-Pro, GPQA, BBH, MuSR) remain weak; a 0.59 B model carries little world knowledge.
- iSarcasmEval, RAGTruth, HLE, CRUXEval and the home-appliance simulator score 0 on the index.
- Single training run, one seed; the learning rate was chosen on a pilot, not tuned on the full run.
- Requires
transformers==5.17(qwen3_5architecture).
License
Apache-2.0 for the weights, head and code in this repository. The Qwen3.5 base model is Apache-2.0. The training datasets keep their own licences.