GemmaDecision-270M v4
A fully tuned Gemma-270M candidate ranker: it scores each supplied choice together with the request, using a scalar decision head. Intended research tasks are banking-support topic selection and three-way evidence relations. It does not generate answers. The complete modified encoder and head are included.
On the private fresh-prompt test, accuracy is 96.50% for four-way banking topics and 78.00% for three-way NLI, with an equal-family mean of 87.25%. Both frozen task targets were met on this private test. See RESULTS.md for both baselines, paired uncertainty, coverage, synthetic challenge results and failures.
Easy Python package
Documentation · Python quickstart · 10 use-case recipes · PydanticAI guide · API reference
Package 0.2.0 uses ONNX Runtime on CPU by default. The standard install requires neither PyTorch, Transformers nor CUDA packages. PydanticAI and the HTTP server are optional extras. Package version 0.2.0 is separate from the frozen model's v0.4.0 version.
The selected ONNX export preserves the frozen weights with lossless compact storage and FP32 computation; its complete runtime download is 573,083,893 bytes (about 573 MB, including the tokenizer), comparable in weight size to the original BF16 checkpoint. It matched the reference's top choice on 700/700 development cases, a conversion-fidelity check rather than 100% decision accuracy; see the runtime validation and CPU measurements.
Install gemmadecision on PyPI and call it directly:
python -m pip install "gemmadecision==0.2.0"
from gemmadecision import decide
team = decide("I was charged twice", choices=["billing", "technical"])
print(team)
The first call downloads the pinned ONNX export and tokenizer, verifies their hashes, and loads the model. Later calls reuse the loaded model in the same Python process; later process starts reuse the downloaded files. Model inference runs locally. No Hugging Face token or external LLM API key is required for these public artifacts. See the deployment guide for explicit offline use.
In a notebook, install into the active kernel's environment:
%pip install --upgrade "gemmadecision==0.2.0"
Restart the kernel or notebook runtime after upgrading, especially if an older GemmaDecision version or its runtime libraries were already imported. Then run the Python example again. Upgrading does not remove dependencies that an older package version installed; the default 0.2.0 inference path uses ONNX Runtime.
For native PydanticAI, install its extra. No server is required:
python -m pip install "gemmadecision[pydantic-ai]==0.2.0"
from typing import Literal
from pydantic_ai import Agent
from gemmadecision import GemmaDecisionModel
agent = Agent(GemmaDecisionModel.local(), output_type=Literal["billing", "technical"])
print(agent.run_sync("I was charged twice").output)
To serve the HTTP API with Rust-powered Granian:
python -m pip install "gemmadecision[serve]==0.2.0"
gemmadecision serve
The server uses the CPU ONNX backend by default. Its local address is
http://127.0.0.1:8700; interactive API documentation is at /docs.
Granian provides Rust HTTP transport; ONNX Runtime performs model computation.
See serving and authentication.
For PyTorch inference on CPU, CUDA or Apple MPS, install the torch extra and
select that backend explicitly:
python -m pip install "gemmadecision[torch]==0.2.0"
from gemmadecision import DecisionEngine
engine = DecisionEngine.from_pretrained(backend="torch", device="cuda")
answer = engine.decide(
"I was charged twice",
{"billing": "Charges and refunds", "technical": "App crashes and sign-in failures"},
)
print(answer.choice)
Use device="cpu" or device="mps" for the corresponding PyTorch device.
A compatible PyTorch build and device are required. For an optional vLLM
server, use a separate supported Linux/CUDA environment:
python -m pip install "gemmadecision[vllm]==0.2.0"
gemmadecision serve --backend vllm --device cuda
The vLLM guide documents its runtime requirements and measured numerical differences. Selecting another backend does not guarantee faster inference or identical scores.
The guides include routing, classification, urgency rubrics, evidence relations, ranking supplied responses, tool recommendations, typed triage and human review. These recipes demonstrate integration patterns, not accuracy guarantees for every application domain.
The original v0.4.0 weights and reference helper remain unchanged. The ONNX runtime export is pinned separately and is derived from that frozen model; runtime validation does not establish new model accuracy or benchmark results. The package's detailed API exposes probabilities derived from raw scores using the previously published fixed temperature. They are not guarantees of correctness. The reference helper below continues to return raw rankings.
Run on CPU
The commands below use the original PyTorch reference helper included with the v0.4.0 model release. They are separate from the default ONNX package path shown above and install that reference helper's own requirements.
From a downloaded complete model directory:
python -m pip install -r requirements.txt
python joint_deployment.py --model . --input example_request.json --device cpu
To download the versioned release and use the Python interface:
import sys
from huggingface_hub import snapshot_download
path = snapshot_download(repo_id='rajan2k/GemmaDecision-270M', revision='v0.4.0')
sys.path.insert(0, path)
from joint_deployment import GemmaDecisionRanker
model = GemmaDecisionRanker(path, device="cpu")
ranking = model.rank(
"The same card payment appears twice on my statement.",
{"payments": "Investigate charges, transfers and refunds.",
"technical": "Investigate app crashes and login errors."},
question="Which support team should handle this request?",
)
print(ranking)
After download, inference uses local files only. AutoModelForCausalLM alone does not apply the decision head. The helper supports CPU, CUDA and MPS selection; only the devices described in the evidence have been tested. Candidates may be a list of texts or a label-to-text mapping, with 2–64 distinct choices. The limits are 2048 state/question tokens and 768 tokens per choice. Long inputs raise an error; they are not silently shortened.
Scores are raw, uncalibrated rankings. They are not probabilities, cannot be compared across requests, and do not establish that the winning choice is correct. The calibration reported in evaluation.json is an evaluation diagnostic; the public rank() interface does not apply it.
What this evaluation means
The banking task supplies the correct topic and three lexical hard alternatives. It is not official 77-way BANKING77 classification or arbitrary tool routing. The evidence task is a class-balanced filtered MultiNLI construction. New prompt groups exclude previously used v2/v3 groups and inputs, but source domains and all banking intents were already encountered. Public foundation-model pretraining overlap is unknown. Synthetic policy challenges are reported separately.
This release makes no JEV/CLM parity, general decision-reliability, or official JevBench claim. Use the error reports and documented task scope when assessing a use case.
Training and provenance
The encoder starts from google/gemma-3-270m revision 9b0cfec892e2bc2afd938c98eabe4e4a7b1e0ca1 and the scalar head starts from the project's v3 joint checkpoint. Both become trainable. Training uses FP32 master weights and AdamW state with BF16 autocast; the chosen encoder is exported in BF16 and evaluated with an FP32 scalar head, normalized last-token pooling and the exact serving input format. CPU serving loads the encoder in FP32. No quantized or unchanged-base substitute is implied.
Checkpoint selection used development data; temperature fitting used separate calibration data; final cases were opened only after freezing the recipe. See base-provenance.json, evidence/frozen-recipe.json, DATA_PROVENANCE.md and REPRODUCIBILITY.md for hashes and limits. Training code is at github.com/rjn32s/GemmaDecision-270M. Exact training reconstruction also needs the documented historical split inputs and initialization head; it is not claimed to work from code alone with one command.
Terms and attribution
The modified model weights are governed by the included Gemma terms and Prohibited Use Policy. Original and vendored Apache code uses LICENSE_CODE; that does not relicense weights or datasets. NOTICE identifies modifications and upstream CLM helpers. Training-source attribution includes direct v4 data and inherited v3 head lineage in RELEASE_ATTRIBUTION_AUDIT.md. Google, CLM and dataset authors do not endorse this project.
JevBench public diagnostic
The frozen v0.4.0 model scored 110/231 (47.62%) on the 231 public JevBench cases, with 195/231 valid outputs. This is an author-run public diagnostic, not an official score or rank. It shows the limits of transfer beyond the banking/NLI development tasks. All 36 refusals exceeded the published 2,048-token state/question limit and count as incorrect in the headline result.
Complete benchmark report, failures and reproducible server.
The adapter converts raw scores with a fixed, previously fitted softmax
temperature; rank() itself remains unchanged. Earlier v0.2 public benchmark
exposure is disclosed in that package. Independent sealed evaluation is being
requested from the maintainers. Weights and the immutable v0.4.0 tag are unchanged.