prabhasa-nyaya-element-judge-4b-arm0
Deployed as production 4B judge 2026-09-30; serving integrity verified; quality claims at constructed tier only. EXPLORATORY. Single seed.
Scope of the 0/225 figure: Constructed in-distribution set, two-judge config C; not real court material. On real court judgments (single-judge, 1 Oct): 11/122 false proofs, CP95 upper 14.5%.
Full fine-tune of Qwen/Qwen3-4B as an element judge for the Nyaya pipeline ("is this legal element ESTABLISHED by these facts?"). Output is a short verdict whose first-token logprobs give p_established.
Training
- Data:
element_judgment_combined.jsonl@36a69ec, SHA-25653697c5762c011b57cf80e00d9ddf6aa5cb54eae948eca8b59c569d7123ab440, 1,463 rows (2,074 balanced train / 136 val). Same file as the deployed 4B v0. - Completion-only loss (
completion_only_loss=True): prompt tokens are masked, only the verdict completion is trained. This is the only intended difference from the deployed 4B, which trained on all tokens. (Masking check:arm0_masking_evidence.json.) - Base:
Qwen/Qwen3-4B, full fine-tune (no LoRA). Base revision1cfa9a72…(1cfa9a7208912126459214e8b04321603b3df60c) is inferred: the training run loaded the unpinned name, this was the only snapshot in the training machine's cache, and the trainer did not record it. The base is not provably pinned. - Hyperparameters: 3 epochs (~390 optimizer steps), batch 4, grad-accum 4, lr 2e-5 cosine, warmup 100,
adamw_bnb_8bit, gradient checkpointing, bf16, max_length 1024, seed default. Identical to the deployed 4B's recipe apart from the loss mask. - TRL 1.13.0, Transformers 5.17.0, PyTorch 2.9.0a0 (NVIDIA 25.09), Datasets 5.0.1, Tokenizers 0.23.2. bf16, single seed.
- Weights file
model.safetensorsSHA-256c2fa5954d5bab3885d13d472e38aad67fe6cbad51f54a08a472fa2f9a6a9eec6.
Results (all from sealed, committed runs; caveats bind every number)
Element level, tau = 0.74, Arm-0 vs the deployed 4B, exact paired McNemar (two-sided):
- Established-side (est200, original fact order): 200/200 vs 183/200 (Arm-0-only 17, deployed-only 0, p 1.5e-5).
- NE-side (ne71 wrong-ESTABLISHED, original order): 6/71 vs 12/71, not significant (p 0.109). Do not read this as Arm-0 beating the deployed 4B on not-established.
- Shuffled fact order on ne71: 7/71 vs 23/71 wrong-ESTABLISHED (p 1.45e-4).
- Arm-0 (completion-only 4B): DEPLOYED to production 2026-09-30 (serving integrity verified: served weights reproduce sealed scores). Config-C on the constructed frozen sets, production fact-id convention: 117/377 Lean-verified proofs vs deployed 105/377 on the same fresh path (12 vs 0 discordant, one-sided exact p=0.00024). Against the earlier 109/377 replay (frozen-logit F_{element_id} convention, not production): 8 vs 0, p=0.0039. False proofs 0/225 all legs (not fresh evidence). Seal-order slip disclosed (log printed before commit); outputs byte-identical on R1's independent re-run. Tier: constructed, in-distribution, single seed. No claim of fewer wrong establishments; transfer to real material untested.
- Config-C = AND of this 4B at tau 0.74, the frozen 32B judge at 0.97, and the Lean registry check (a structural check, not proof verification).
Limitations
- The 377/225/271 evaluation sets are disjoint from the training file by id and text, but in-distribution: same task family, same generators. No transfer or real-material claim.
- One seed, one shuffle, no multiplicity correction; units within a contract are not independent.
- The Lean step in config-C is a structural check, not a proof verification.
- Not evaluated for use outside this pipeline's prompt format; serving requires the pinned prompt builder and
top_logprobs. - Transfer to other data is untested; no not-established superiority claim.
- Serving-switch staging and a rollback path are pre-registered separately; this repo is the artifact only.
Files
Weights, config, tokenizer and chat template as trained. training_args.bin is the TRL arguments pickle; do not load it untrusted.