Pivot
Fast closed-set decisions from context and candidate actions
Pivot is a 357.6M-parameter bidirectional decision encoder. Give it a context and two or more candidate answers; it scores the set in one forward pass and returns the chosen answer, its position, and a probability for every candidate. It does not generate free-form text. The full FP32 checkpoint, tokenizer and Transformers custom runtime are stored in this model repository.
| Measured result | Pivot |
|---|---|
| JevBench v1.4.1 public accuracy | 46.32% (107 / 231) |
| NVIDIA H200 warm single-decision p50 / p95 | 15.8 / 19.9 ms |
| NVIDIA H200 throughput, batch 32 | 545.3 decisions/s |
| 4-thread Xeon CPU warm single-decision p50 / p95 | 797.6 / 1,089.0 ms |
| 4-thread Xeon CPU throughput, batch 4 | 3.77 decisions/s |
These are measurements on the pinned checkpoint, in FP32, including tokenization and scoring. GPU and CPU throughput used different batch sizes. The 46.32% figure is public-task accuracy; the official JevBench v1.4 composite score has not been measured because the sealed/judge portion and cost input are unavailable. Protocol and limitations · Full measurement details
Start with one decision
Install a suitable PyTorch build and the runtime packages:
python -m pip install "transformers==5.17.0" "safetensors==0.8.0"
import torch
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Q1z/Pivot", trust_remote_code=True)
model = AutoModel.from_pretrained(
"Q1z/Pivot", trust_remote_code=True, dtype=torch.float32
).eval()
context = "CONTEXT:\nA customer disputes an invoice and asks for a correction."
options = [
"route to billing support",
"route to technical support",
"route to sales",
]
decision = model.choose(tokenizer, context, options)
print(decision) # {"choice": ..., "index": ..., "probs": [...]}
The serving configuration now defaults to 512 context tokens and 128 option tokens, matching the published public evaluation. It does not change the checkpoint weights. Run on CUDA with model.to("cuda") if your PyTorch build supports it. Review the repository's custom model code before enabling trust_remote_code=True.
Three ways to use Pivot
| Method | Input | Output |
|---|---|---|
model.choose(tokenizer, context, options) | Context and ordered answer strings | Chosen answer, index, probability vector |
model.decide_native(tokenizer, context, candidates) | Candidate IDs, semantic text, optional abstain action | Selected ID, relative confidence, per-candidate probabilities |
model.decide(tokenizer, state, questions) | Several typed choice / yes-no / score questions | Typed decision response |
For repeated decisions using the same options, encode_candidates, encode_context, and choose_cached reuse candidate representations. This can avoid repeated candidate encoding; the measured throughput above uses the uncached path. Examples: basic, structured decisions, cached candidates, and full inference guide.
Pivot's probabilities are relative to the supplied candidate set. Give each candidate a clear, distinct meaning. An abstain route must be an explicit candidate; a high relative probability alone does not establish real-world correctness or safety.
Evaluated performance
The official JevBench v1.4.1 public tasks were scored with the official per-task scorer at commit 24b9b5c1609a7a9e8fa14f49e5985a836c9dc842. The exact evaluated model commit was 14bf8c26bf344ebdf88e22a4b6152dc5f75f3578.
| Public tier | Correct / tasks | Accuracy | ECE, 10 bins |
|---|---|---|---|
| Original | 27 / 72 | 37.50% | 0.525 |
| Easy | 39 / 48 | 81.25% | 0.114 |
| Hard | 41 / 111 | 36.94% | 0.379 |
| All public tasks | 107 / 231 | 46.32% | — |
The frozen evaluation format is a CONTEXT: prefix, task rubric descriptions where provided (otherwise humanized labels) in official label order, right truncation, 512 context tokens and 128 option tokens. Public accuracy is not an official leaderboard score. Reproduce public results or run the CPU speed entry point. The CPU notebook provides an interactive alternative.
Repository guide
- Inference methods and examples
- Serving interfaces and request formats
- JevBench protocol, revision, and score scope
- Results, hardware, charts, and caveats
- Reproducible public benchmark script
- Standalone CPU speed script
- Pinned result files and chart
- What changed in this update
The files model.safetensors, modeling_pivot.py, modeling_lfm2_bidirectional.py, pivot_model.py and pivot_infer.py contain the checkpoint and model runtime. Package manifest records file hashes. The original checkpoint and evaluated metrics remain anchored to the exact revision above.