Jev-alpha-26B-A4B

A standalone decision model derived from Google Gemma 4 26B A4B IT. It reads a state and typed questions, then exposes probabilities over the allowed options. This repository contains full merged BF16 weights. It does not require a LoRA adapter at inference. The model retains the upstream Gemma4ForConditionalGeneration architecture and tokenizer. Only text decision behavior is fine-tuned and evaluated; other modalities and general chat are not validated here.

Checkpoint and training

This is seed 42, KL weight 0.1, checkpoint 2100. It is not an average of three adapters. The development screen selects the KL weight. The final test does not select this seed. Training uses 8,400 questions, two epochs, LoRA rank 8, alpha 16, and dropout zero. The loss is option-token cross-entropy plus 0.1 times conditional reference-to-model KL. The reference top 64 tokens exclude the question's valid option tokens at the answer position. Base revision: 4d7ae4984b7db7de8f8457170b3f1a419ee76d52. The 26B/A4B name follows the upstream size class; all base weights remain required in memory.

Measured standalone release

These measurements reload the saved merged model with Transformers, without PEFT or an adapter. Equal-type final accuracy: 79.50% on 1,100 questions. Each source contributes 100 final questions. The aggregate gives equal weight to noul, choice, and score. The merged model has its own fitted temperature: 1.736507, from 300 separate calibration questions. BF16 merging changes numerical behavior. Use these release results instead of the earlier adapter or three-seed mean. Calibration error uses ten equal-width confidence bins. Lower Brier and calibration error are better.

Full aggregate metrics and merge checks are in release_metrics.json.

Comparison with published Jev results

The blue line uses this downloadable merged seed-42 model, not the three-seed average from the GitHub README. Jev values are third-party measurements published by Kev. The question samples, prompts, and option construction differ. This is a descriptive comparison, not a paired benchmark or a claim of parity. Jev's training data is unknown. Our final test uses 100 questions per dataset from training source families; the Jev samples contain 80 or 150 questions. The radar uses a 0–100% scale. Its area is not an aggregate metric.

Sources at Kev commit fe64b1274ea7f80d4095866df90666abb03e9cf6:

• Transfer-v4: MMLU, SciQ, QNLI, PAWS.
• Decision-v4: BoolQ, DBpedia, SST-5, Yelp.
• Breadth-v1: CLINC and HellaSwag.
• Devtools-v1: When2Call.

Exact plot values and source hashes accompany the image.

Use with Transformers

Tested with Transformers 5.16.1, PyTorch 2.8.0, BF16, SDPA, and grouped_mm on an H200. Different precision or expert backends can change probabilities and calibration. Install the Jev-alpha package for its shared renderer. The example uses direct logits and never calls generate. A generic text-generation pipeline does not return this decision schema.

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from jev_alpha.data import render, prompt_ids, label_ids

repo = "chenz53/Jev-alpha-26B-A4B"
model = Gemma4ForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa").eval()
model.set_experts_implementation("grouped_mm")
tokenizer = AutoProcessor.from_pretrained(repo).tokenizer
with open(hf_hub_download(repo, "temperature.json")) as stream:
    temperature = json.load(stream)["temperature"]
record = {"state": "The payment failed. The login works.", "questions": {"route": {
    "type": "choice", "instructions": "Which team handles the problem?",
    "criteria": {"payments": "Payment failures", "account": "Login access"}}}}
messages, order = render(record, "route")
ids = torch.tensor([prompt_ids(tokenizer, messages)], device=model.get_input_embeddings().weight.device)
with torch.inference_mode():
    logits = model(input_ids=ids, attention_mask=torch.ones_like(ids), logits_to_keep=1, use_cache=False).logits
scores = logits[0, -1, label_ids(tokenizer)[:len(order)]].float()
probs = (scores / temperature).softmax(-1).cpu().tolist()
print(dict(zip(order, probs)))

Use no/yes options for noul and numeric level keys for score. Each question needs 2–26 options. The evaluated context limit is 4,096 tokens. Each question repeats the state in its own forward pass.

Data and limits

Training uses 6,720 public questions and 1,680 directly authored questions, with 2,800 questions per type. Sources include BoolQ, QNLI, PAWS, MMLU, SciQ, HellaSwag, CLINC, DBpedia, SST-5, Yelp, and When2Call. Final controls come from the same source families, not unseen datasets. The BoolQ controls use unused cleaned training records. The When2Call public training subset contains only clarification and cannot-answer targets; authored questions include other actions. New authored questions and action labels lack independent human review. These limits can affect action selection. This release does not establish Jev-equivalent speed, cost, or preservation of all foundation-model abilities. See dataset sources and terms. SciQ carries CC-BY-NC-3.0 terms; other sources have their own terms. The model license does not grant rights to the source datasets. No source dataset records are included here.

License and attribution

Gemma 4 uses Apache 2.0, not the older Gemma Terms of Use. The release includes LICENSE, NOTICE, and the upstream BASE_MODEL_CARD.md with Google DeepMind attribution. Apache 2.0 does not require a Gemma prefix in the derivative name. This project does not claim Google or Jev endorsement. Project: https://github.com/chenz53/Jev-alpha