qwen3.5-0.8b-nvsh-tool-jev-scorer-v2
A fine-tune of Qwen/Qwen3.5-0.8B (revision
2fc06364715b967f1860aea9cf38778875588b17) for nvsh's Tier 2 as a Jev-style
candidate scorer (Track B): every candidate -- each operation in nvsh's table,
plus explain and escalate -- is listed in the prompt under a one-letter
label, and the model's next-token log-probabilities over those labels are read
once. The highest-scoring label is the choice; an operation's arguments come
from nvsh's deterministic grounding, never from the model. It is not a
generative tool caller. Training run scorer-r3b.
This is a derivative work based on the base model. The weights were changed by
LoRA fine-tuning and merged; the tokenizer and chat template are the base
model's, unchanged. See NOTICE.
config.json: mtp_num_hidden_layers is set from 1 to 0 in this upload.
The checkpoint is text-only and its weights hold no mtp.* tensor, so the
multi-token-prediction head the base's config declares does not exist here.
At a glance
What it does. A 0.8B model that decides, for a request typed at an
nvsh shell on an NVIDIA Jetson or DGX
Spark, which of nvsh's typed operations to propose -- or to explain, or to
escalate to a bigger model when none fits. nvsh lists the offered candidates
under one-letter labels; the model's next-token probabilities over those
labels are read once (no text is generated), calibrated, and passed through a
small gate that proposes, explains, escalates or abstains. Arguments are never
produced by the model: nvsh grounds them deterministically. It is a
scorer, not a chat model or a free-form tool caller, and nvsh always asks
the human before running anything it proposes.
What we have.
| Repository | Contents |
|---|---|
qwen3.5-0.8b-nvsh-tool-jev-scorer-v2 | bf16 merged fine-tune (safetensors), tokenizer, chat template |
qwen3.5-0.8b-nvsh-tool-jev-scorer-v2-gguf | Q4_K_M GGUF (the deployed build), plus calibration.json (temperature 1.5366) and gate.json (read-only margin 0.2) fitted on that build |
The operation table it was trained on has 16 operations (status and stats
reads such as gpu_stats, service_status, disk_stats; and mutating
service_restart, container_restart, power_set). nvsh's runtime does not
load this scorer yet; wiring it into the tier stack is
nvsh#69.
How it was tested. Only on sides never used to train or choose it:
- the issue-53 test set (198 requests: 83 expect an operation, 79 an escalation, 36 an explanation) and its missing-candidate slice (83 requests where the right operation was removed from the offered list, so escalating is the only right answer);
- a sealed held-out set (149 requests, never read by the development side) and its missing-candidate slice (60);
- on a DGX Spark and on an AGX Orin, deployed
Q4_K_Mwith the frozen calibration and gate; - a label-permutation probe (9,900 trials: option order, letters, subsets, paraphrase);
- nvsh's DeepEval release gate (nvsh#64), against 15 frontier and open reference models on the same test cases.
What passed and what missed (pre-registered bars, deployed Q4_K_M with
calibration and gate):
| Bar | Target | Test (198) | Held-out (149) | Met |
|---|---|---|---|---|
| No wrong mutating action (incl. missing-candidate slices) | 0 | 0 | 0 | yes |
| Calibration, ECE | <= 0.10 | 0.016 (Orin 0.009) | 0.044 (Orin 0.046) | yes |
| Complete served readouts | >= 95% | 198 / 198 | 149 / 149 | yes |
| Label-permutation answer change, pooled | <= 2.3% | 2.07% | not measured | yes |
| Uncertainty kept apart from escalation | separate outcome | yes | yes | yes |
| Missing-candidate escalation | >= 80% | 85.5% (Orin 84.3%) | 76.7% (46 / 60) | no (held-out) |
Other figures: right proposals 79 / 83 on test and 49 / 60 on held-out; escalation recall 94.9% (test) and 97.1% (held-out); warm decision latency 124 ms on a DGX Spark and 355 ms on an AGX Orin.
Compared with other models (nvsh DeepEval gate, run 8b59d0afa150,
2026-09-29, test set, references answering the same offered-candidate
question):
| Model | Right | Escalation recall | Wrong mutating | Missing-candidate escalation | Missing-candidate wrong mutating |
|---|---|---|---|---|---|
this model (Q4_K_M, calibration + gate) | 79 / 83 | 94.9% | 0 | 85.5% | 0 |
| gpt-6-sol | 81 / 83 | 72.2% | 0 | 65.1% | 0 |
| claude-opus-5-5 | 82 / 83 | 67.1% | 0 | 61.4% | 0 |
| deepseek-v4.1-flash | 82 / 83 | 67.1% | 0 | 65.1% | 0 |
| gpt-6-luna | 82 / 83 | 63.3% | 0 | 57.8% | 0 |
| qwen3.8-max | 82 / 83 | 62.0% | 0 | 48.2% | 1 |
The references pick the right operation as often or slightly more often; this
model is far better at knowing when not to act. Full report: nvsh
PR #72
(docs/benchmarks/2026-09-29-deepeval-gate-run-8b59d0afa150.md); the
calibration cycle report is
docs/benchmarks/2026-09-25-tool-jev-calibration-cycle.md.
Known limitations.
- The missed bar: on the held-out missing-candidate slice, 14 of 60 requests were not escalated; the model proposed a related read-only operation instead (for example a GPU check when the memory check was removed). None was a mutating action.
- Held-out right proposals are lower than test (81.7% vs 95.2%).
- The held-out read-only slice is over-confident (ECE 0.147 on 49 entries) although the side as a whole is calibrated; threshold read-only proposals with that in mind.
- Scope is nvsh's own operation table, English requests, Jetson / DGX Spark context. It is not a general tool-use model, and its figures say nothing about other tools or prompts.
About the files. The calibration and gate above were fitted on the
Q4_K_M build and ship only with the GGUF repository; the Q4_K_M readout
differs from this bf16 checkpoint on some entries, so fit your own
temperature if you serve bf16. This checkpoint's own uncalibrated figures are
in the Results tables below (test: 79 / 83 right, 0 wrong mutating, ECE 0.023).
Licence
This model is a derivative of the base model under the Apache License 2.0.
The LICENSE file is included. The base model is
Qwen/Qwen3.5-0.8B
(revision 2fc06364715b967f1860aea9cf38778875588b17).
Use
nvsh's runtime does not load a candidate scorer. nvsh's
scripts/lfm-finetune/scorer.py reads it: one prompt lists every candidate
under a one-letter label and the next-token log-probabilities over those
labels are read once (served by vLLM it needs --max-logprobs 20000).
Results
Measured with nvsh's scripts/lfm-finetune/measure.py. Each table is the
"Issue 46 metrics" section of the named report, quoted as written; the
column header is the name the model was served under.
Tier 2 measurement, 2026-09-25: final-scorer-r3b-exact
From 2026-09-25-lfm-final-scorer-r3b-exact.md:
| Metric | scorer-r3b |
|---|---|
| Right proposals (metrics.py) | 79 of 83 |
| Right proposals, 95% bootstrap CI | [90.4%, 98.8%] (n=83) |
| Abstention recall (escalate entries escalated) | 97.5% (77 of 79) |
| Abstention recall, 95% bootstrap CI | [93.7%, 100.0%] (n=79) |
| Abstention precision, strict (deviation d2) | 98.7% |
| Abstention precision, strict, 95% bootstrap CI | [96.2%, 100.0%] (n=198) |
| False-positive tool calls (proposals on explain/escalate entries) | 1 of 115 |
| False-positive tool calls, 95% bootstrap CI | [0.0%, 2.6%] (n=115) |
| Wrong mutating, total (wrong operation + wrong arguments) | 0 (0 + 0) |
| Invalid outputs | 2 of 198 (not_grounded: 2) |
| Invalid outputs, 95% bootstrap CI | [0.0%, 2.5%] (n=198) |
| Lines with a candidate distribution | 198 of 198 |
| Scorer readouts, complete / incomplete (never renormalised) | 198 / 0 |
| ECE (10 equal-width bins) | 0.023 |
| ECE, 95% bootstrap CI | [0.006, 0.046] (n=198) |
| Brier (multi-class) | 0.047 |
| Brier, 95% bootstrap CI | [0.012, 0.090] (n=198) |
| ECE / Brier before --calibration | n/a |
| Tokens generated per decision, mean / median | 0.0 / 0.0 |
| Time to first decision, cold / warm median / warm p95 | 641 ms / 83 ms / 87 ms |
| Decision latency, cold / warm median / warm p95 | 641 ms / 83 ms / 87 ms |
| Non-empty think blocks (must be 0) | 0 |
Tier 2 measurement, 2026-09-25: final-scorer-r3b-missing-candidate-exact
From 2026-09-25-lfm-final-scorer-r3b-missing-candidate-exact.md:
| Metric | scorer-r3b |
|---|---|
| Right proposals (metrics.py) | 0 of 0 |
| Right proposals, 95% bootstrap CI | n/a (n=0) |
| Abstention recall (escalate entries escalated) | 81.9% (68 of 83) |
| Abstention recall, 95% bootstrap CI | [73.5%, 89.2%] (n=83) |
| Abstention precision, strict (deviation d2) | 100.0% |
| Abstention precision, strict, 95% bootstrap CI | [100.0%, 100.0%] (n=83) |
| False-positive tool calls (proposals on explain/escalate entries) | 12 of 83 |
| False-positive tool calls, 95% bootstrap CI | [7.2%, 22.9%] (n=83) |
| Wrong mutating, total (wrong operation + wrong arguments) | 0 (0 + 0) |
| Invalid outputs | 1 of 83 (not_grounded: 1) |
| Invalid outputs, 95% bootstrap CI | [0.0%, 3.6%] (n=83) |
| Lines with a candidate distribution | 83 of 83 |
| Scorer readouts, complete / incomplete (never renormalised) | 83 / 0 |
| ECE (10 equal-width bins) | 0.141 |
| ECE, 95% bootstrap CI | [0.080, 0.224] (n=83) |
| Brier (multi-class) | 0.295 |
| Brier, 95% bootstrap CI | [0.172, 0.435] (n=83) |
| ECE / Brier before --calibration | n/a |
| Tokens generated per decision, mean / median | 0.0 / 0.0 |
| Time to first decision, cold / warm median / warm p95 | 618 ms / 82 ms / 85 ms |
| Decision latency, cold / warm median / warm p95 | 618 ms / 82 ms / 85 ms |
| Non-empty think blocks (must be 0) | 0 |
Training data
2,503 candidate-scorer training rows: 2,191 train records (431 hand-written corpus requests, 75 fresh-drafted train-side requests, 491 targeted supplement entries and 1,194 reviewed variations) plus 312 missing-candidate rows, each row with its own offered candidates and letter map; written and reviewed by Apache-2.0 teachers, frozen before training, leakage-checked against validation, test and two sealed held-out sets
Requests were rewritten into variations by a local pipeline
(scripts/lfm-finetune/augment.py). Two reviewer models were each asked whether the rewrite still calls for exactly that answer: reviewer B's verdict alone decided whether a rewrite was kept; reviewer A's verdict, where it was asked, was recorded as advisory and did not decide. Deterministic guards also refused any variation naming
an operation identifier or copying answer wording. A training record that repeats a
validation or test entry is dropped (merge_variations.py --exclude), and the
test side is never used to choose a run.
| Model | Licence | Role |
|---|---|---|
| Qwen 3.6 35B-A3B | Apache-2.0 | wrote the variation |
| Qwen 3.8 27B | Apache-2.0 | copyedited it |
| Gemma 4 26B-A4B | Apache-2.0 | reviewer A, advisory: asked and recorded, did not decide |
| Qwen 3.8 27B | Apache-2.0 | accepted it (reviewer B, deciding) |
Reviewer B is also the corrector in this run (Qwen 3.8 27B): its verdict is not independent of the copyedit it made.
The teachers' licences do not carry over to their outputs.
Recipe
scripts/lfm-finetune/pipeline.sh and docs/lfm-finetune.md and
docs/qwen-tool-jev-finetune.md in the nvsh
repository: seeded split, augmentation, scripts/lfm-finetune/train_scorer.py (label-restricted
cross-entropy LoRA training),
merge, and measurement against the stock model.