Nirnaya (निर्णय)

Nirnaya is a Loggenix-derived 0.4B/0.2B-active MoE trained to predict typed decision distributions over ordered options.

Typed Decisions evaluation at a glance

This is a fine-tuned specialist result, not a zero-shot score. It was self-reported on the benchmark's 400-case test split; the benchmark maintainer has not independently rerun it. The benchmark eval.yaml task IDs are accuracy, kl_from_gold, brier, and ece, and our .eval_results/typed-decisions.yaml uses those same IDs. See the evaluation report and reproduction artifacts for the split, prompt, scoring, and latency details. The benchmark card keeps fitted/fine-tuned results in a separate table from zero-shot models.

Training

• Base: kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s4
• Dataset: kshitijthakkar/nirnaya-probability-distill-v1, pinned at 84c269a6008d59e9c9b49f5a93f1605f0f8bafb6.
• Full-parameter fine-tuning for one epoch on the train split; dev and calibration splits were held out.
• Soft-label cross-entropy plus a 0.20-weighted Brier loss over the valid options.
• Learning rate 2e-4, warmup ratio 0.10, packed 16,384-token context, bf16, gradient checkpointing, and FlashAttention 2.
• Weights & Biases tracking was required for the run (nirnaya-softdistill).

Typed probability inference

The tokenizer includes <|nirnaya_option_0|> through <|nirnaya_option_173|>. For a question with k ordered options, compute next-token logits at the assistant generation position, retain the logits for option tokens 0 through k-1, and softmax those logits to obtain the probability vector aligned with the supplied options. The option ordering must match the prompt ordering.

Run locally with Transformers

Install the runtime dependencies:

pip install torch transformers accelerate

Save this as nirnaya_infer.py. It follows the model's training prompt format and reads the model's option-token logits directly, so it returns a normalized probability distribution without generating explanatory prose.

import json
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_ID = "kshitijthakkar/nirnaya-0.4b-0.2a-decision"
SYSTEM = (
    "You are Nirnaya, a typed decision model. Read the state, question, "
    "criteria, and ordered options. Assign a probability to each option. "
    "The output is a distribution over the options, not an explanation."
)


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
).eval()


@torch.inference_mode()
def decide(*, state, question_type, question, options, criteria=None):
    if not 2 <= len(options) <= 174:
        raise ValueError("Nirnaya supports 2 to 174 ordered options")


    # Reset peak statistics for this call. Synchronize first so prior GPU work
    # is not included in this call's latency or memory window.
    if torch.cuda.is_available():
        torch.cuda.synchronize()
        torch.cuda.reset_peak_memory_stats()
    start_time = time.perf_counter()


    criteria_text = json.dumps(criteria or {}, ensure_ascii=False)
    choices = "\n".join(f"{i}: {option}" for i, option in enumerate(options))
    user = (
        f"Question type: {question_type}\n"
        f"Question: {question}\n"
        f"Criteria: {criteria_text}\n"
        f"Options (in target probability order):\n{choices}\n"
        f"State:\n{state}"
    )
    prompt = tokenizer.apply_chat_template(
        [{"role": "system", "content": SYSTEM},
         {"role": "user", "content": user}],
        tokenize=False,
        add_generation_prompt=True,
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    next_logits = model(**inputs).logits[0, -1].float()


    tokens = [f"<|nirnaya_option_{i}|>" for i in range(len(options))]
    token_ids = [tokenizer.convert_tokens_to_ids(token) for token in tokens]
    if any(tokenizer.convert_ids_to_tokens(tid) != token
           for token, tid in zip(tokens, token_ids)):
        raise RuntimeError("Tokenizer is missing a Nirnaya option token")
    probabilities = torch.softmax(next_logits[token_ids], dim=0).cpu().tolist()


    # CUDA kernels run asynchronously; synchronize before stopping the timer.
    if torch.cuda.is_available():
        torch.cuda.synchronize()
    latency_ms = (time.perf_counter() - start_time) * 1000
    peak_vram_mb = (
        torch.cuda.max_memory_allocated() / (1024 * 1024)
        if torch.cuda.is_available() else 0.0
    )
    print(
        f"[{question_type.upper()}] Latency: {latency_ms:.2f} ms | "
        f"Peak VRAM: {peak_vram_mb:.2f} MB"
    )


    return {
        "type": question_type,
        "probabilities": dict(zip(options, probabilities)),
    }


state = "A checkout has failed twice with HTTP 503 from the payment API."


# Choice: options are the possible actions.
choice = decide(
    state=state,
    question_type="choice",
    question="What should the on-call engineer do next?",
    options=["retry with backoff", "page the payment service owner", "ignore"],
    criteria={"prefer": "safe recovery and timely escalation"},
)


# Score: supply score labels in the order whose probabilities you want.
score = decide(
    state=state,
    question_type="score",
    question="How severe is this incident?",
    options=["1", "2", "3", "4", "5"],
    criteria={"scale": "1=low, 5=critical"},
)


# Noul: the training data uses this fixed false/true option order.
noul = decide(
    state=state,
    question_type="noul",
    question="Is this incident currently affecting payment requests?",
    options=["false", "true"],
)


print(json.dumps({"choice": choice, "score": score, "noul": noul}, indent=2))

Latency covers prompt construction, tokenization, model inference, and probability conversion. Peak VRAM is PyTorch's peak allocated memory for the current CUDA device during the call; it includes the loaded model weights.

Run it with:

python nirnaya_infer.py

The returned probabilities are normalized over the options supplied for that question. Nirnaya was trained on Noul as a soft distribution over the ordered classes false, true; the example preserves that order and returns both class probabilities. Keep option labels and order aligned with the model's training convention when comparing results.

Internal JevBench comparison

We evaluated Nirnaya and Laya on the same 231 public JevBench v1.2 tasks on NVIDIA L4 GPUs on 7 October 2026. The Jev reference uses the published jev-1.13.0 per-task outcomes from the JevBench public artifact (artifact revision v1.3.0) on those exact task IDs.

The first Nirnaya path returns the full option probability distribution. Its latency reflects vLLM raw full-vocabulary log-probability retrieval. The fast Nirnaya path returns only the highest-scoring option and does not provide a calibrated probability distribution; its latency is not directly comparable to the full-probability path. Laya was run serially, while Nirnaya used vLLM batching, so these timings describe these tested configurations rather than a controlled throughput comparison. The frozen JevBench public artifact lists Laya at 135/231 (58.4%); our live Laya re-run scored 134/231.

This is a comparison on the public subset, not the official full-board composite score and not an evaluation on sealed cases. Evaluated Nirnaya checkpoint revision: 0231b2d33e5eda7edd4dd0f0be683b4971b9921b.

LocalLLaMA Typed Decisions evaluation

Nirnaya was evaluated on the held-out all/test split of LocalLLaMA/typed-decisions. This is a fine-tuned specialist result, so compare it with fitted specialists rather than the zero-shot table. The upstream all/train split contains 6,000 rows; our training-data manifest records 5,735 in the training split, 135 in dev, and 130 in calibration. The all/test split was excluded from training.

Download the interactive Plotly chart (.html) · Evaluation report and reproduction artifacts

The evaluation covered 400 cases / 2,000 decisions with zero inference errors. Each case adapter call received the shared state and all five questions; it then scored the five training-compatible per-question prompts together in one padded model forward pass on an NVIDIA L4. The questions are not jointly conditioned on each other. Model loading is excluded. Case latency was 45.62 ms p50, 51.26 ms p95, and 46.45 ms mean. This is a self-reported local L4 measurement, not a controlled comparison with leaderboard latency submissions or an independent maintainer reproduction.

ECE uses 10 equal-width bins and compares top-label confidence with argmax agreement. The Brier score is the sum of squared probability errors across options. Evaluated model revision: adcca573db021c43718984baa911b47bdfd045df. These are self-reported results pending benchmark-maintainer review.

Data and limitations

The dataset has 1,469,057 training decisions, 30,578 dev decisions, and 30,651 calibration decisions. It contains source-traceable probability targets from Apache-licensed Jev-style datasets. Nirnaya is not an official Jev model. Evaluate typed accuracy and probability calibration on held-out, non-overlapping data before relying on outputs.

Evaluation status

Completed the official Decision Index 0.3 public suite on this pinned checkpoint. Public Decision Index: 1.60 (balanced raw 24.11); 140,178/140,178 scoreable requests recorded, including 3 unsupported request(s) from the declared 174-option capacity. The public index is only the public component; maintainers calculate the Full score from their private tests.

Reported request latency: {"mean": 26.8, "median": 30.3, "p95": 71.0}. Maintainers measure latency independently on an RTX PRO 6000.

Artifacts, suite verification, environment, calibration report, and adapter: kshitijthakkar/nirnaya-decision-index-0.3-results. Model revision: adcca573db021c43718984baa911b47bdfd045df; reproduction-kit commit: 9eb2dbe2a358004c8782c66e40a83ac07b953fec.