Qwen3.8-9B Hermes FC — Clean Restraint (v29 A/B, arm B, experimental)
Addendum 2026-09-03 — evaluation-methodology correction
Read this before you use the numbers below.
After this checkpoint was published, we tested the base model
(Empero/Qwen3.8-9B-Distill, no fine-tuning) through the live Hermes
interface: the OpenAI-native tools parameter path that Hermes actually
uses. The base model calls tools correctly on its own.
The harness that certified this checkpoint did not use that interface.
It baked tool schemas into the system text, never passed tools, and
required raw <tool_call> XML in the response. That prompt shape matches
neither the base model's own chat template nor the live Hermes request
path. Every gate score in the v26-to-v29 lineage measured this synthetic
interface.
The statement in this README that "the base distill contributes zero tool-calling" is incorrect. It measured the synthetic interface.
Corrected comparison (native tools path, same 4-tool battery, greedy):
| model | tier-1 fired | tier-1 name=todo | tier-2 todo-first | tier-3 fired (want 0) | tier-4 fired (want 0) |
|---|---|---|---|---|---|
| base (no fine-tune) | 18/20 | 17/20 | 3/10 | 5/10 | 0/10 |
| v28 ckpt-175 (this model) | 17/20 | 10/20 | 8/10 | 9/10 | 4/10 |
| armB2-175 (clean retrain) | 10/20 | 8/20 | 10/10 | 4/10 | 0/10 |
What the fine-tune lineage actually changed: todo-first planning on multi-step tasks (tier-2 name=todo 3/10 base to 8/10 v28 to 10/10 armB2). That is real, and it is the one capability the lineage added. The costs: restraint regressions (tier-3 over-trigger 5/10 to 9/10; tier-4 tool-drift on trivia 0/10 to 4/10) and, in armB2, over-suppression (tier-1 10/20).
Status: the fine-tune program is frozen as of 2026-09-03. The base model is the author's runtime default. This checkpoint stays published as the strongest measured demonstration of SFT-taught todo-first planning in the lineage, and as the artifact that exposed the harness error. The Evaluation section below describes behavior on the synthetic interface; do not read it as native-interface competence.
Reproduction: scripts/native_battery.py + scripts/todo_probes.jsonl
scripts/no_drift_probes.jsonlproduce the corrected table (any GGUF,CTX=65536to match the serve config).
- Base model: Empero/Qwen3.8-9B (Apache-2.0)
- Warm start: v27 merged weights (
qwen3.8-9b-hermes-fc-todo), QLoRA adapter on top (envelope + tool tokens already baked in) - Training data: 218 trajectories (ShareGPT format, 551 todo calls),
generated by GLM-5.3-Flash via the HuggingFace router on 112 synthetic
tasks mined from the author's agent corpus — not the author's own traces:
- 190 teacher trajectories (85% verified by sandbox execution; failures with recovery injected for 15.8% of rows)
- 61 rows carry distilled short-chain-of-thought; 159 envelope-only
- 12 synthetic web-search tasks (first web signal)
- Full dataset published separately (see
hermes-fc-v28-data); probes were kept virgin (zero overlap with generation or training)
- Method: QLoRA 4-bit, rank 16 / alpha 16, dropout 0, batch 1 x grad-accum 8 (effective 8), lr 2e-4, warmup 0.1, MAX_SEQ 6144, 8 epochs = 214 steps (final loss 0.02085, avg train loss 0.09405)
- Vocabulary: 248077 → 248079 (2 added tool tokens:
<|tool_call|>,<|tool_response|>) - Weights: BF16 full merge (13 shards, 18.9 GB, 9.20B params) plus a ROCmFPX quantized GGUF (see companion repo)
- Architecture note: merge preserves the 15-key MTP (multi-token
prediction) head from the base —
Qwen3_5ForConditionalGenerationwithtext_config.vocab_size = 248079. Visual tower dropped (text-only).
Evaluation (50-probe battery, greedy, @ Q4 64k ctx)
Tiers: T1 = 20 real DJLougen todo traces held out of ALL training generations (GT = first todo call). T2 = 10 synthetic multi-step tasks → autonomous todo reach-for. T3 = 10 trivial asks (want 0/10) → over-trigger guard. T4 = 10 off-topic chat asks (want 0 fired, 0 topic drift into fine-tune vocabulary like rocm/quant/llama-swap).
| tier | v28-175 (control, warm) | armB2-175 (this model) |
|---|---|---|
| T1 fired | 17/20 | 6/20 |
| T2 fired / fmt-exact | 8/10 / 4/10 | 10/10 / 10/10 |
| T3 fired (want 0) | 5/10 | 1/10 |
| T4 fired / drift (want 0/0) | 2/10 / 0 | 0/10 / 0 |
What this model proves (the experiment's result):
- Contamination is real and this arm fixes it. v28-175 fires on 2 of 10
off-topic chat probes (learned tool reflex from ~1900 rocm-mention rows
in its warm lineage); armB2 fires 0/10 with zero topic drift. Clean base
- filtered corpus = no tool reflex on chat.
- Restraint works. T3 1/10 vs v28's 5/10 — the 87 no-call rows teach when NOT to reach for a tool.
- When it fires, it is clean. T2 10/10 with 10/10 format-exact (bare envelopes, no prose wrap) — the best format score in the lineage.
- Perplexity on a held-out probe ruler: Q4 1.148 vs Q8 1.145 (the published Q4 quant is effectively lossless).
Known limitation (why this is experimental — read this)
Tier-1 collapse on assistant-final transcripts. When a conversation
ends with an assistant narration that in the real trace continued into a
todo call, this model emits nothing (<|im_end|> immediately) instead of
continuing. 13 of 20 T1 probes end that way; all 13 fail. Root cause
(measured): 37% of its training rows end on bare narration (the restraint
rows teach "answer in text, then stop"), and 12 of those 20 probes also
carry the harness cap message ("maximum iterations… without calling any
more tools") — this model obeys it, where v28-175 ignores it and fires
anyway.
Live impact: on fresh turns (user ask or tool result → model's next turn) it calls correctly — T2's 10/10 is that shape. The stall only appears when the model is asked to continue from its own prior narration (real traces cut mid-flow, or a truncated continuation).
Fix in training: v29 armB2-r4 adds fused narration+call rows and cap-message rows ending in a proper text summary. When it lands it will be published as the follow-up; this checkpoint exists so the restraint / contamination findings are testable now.
Do not use this as a drop-in replacement for the current default in
long agentic sessions without the continuation fix. For general tool use
today, prefer qwen3.8-9b-hermes-fc-tooluse (v28). Use this model to test
restraint, contamination-free chat, and format cleanliness.
Intended use
Research artifact for experimenters working on tool-call behavior in
9B-class models — not a product. The tool_call XML envelope and tool
schemas match the Hermes agent runtime this was trained on. v29 is an
ongoing A/B (arm A = v28-175 control, arm B = this clean-base treatment,
arm B2 = this checkpoint's retrain lineage).
What this repo contains
- The BF16 merged weights (this repo)
- All scripts that produced the model and dataset (
scripts/)
Reproducibility
All scripts that produced this model are in scripts/:
| file | purpose |
|---|---|
build_v26.py | shared helpers (build_system, pad_to_bucket, conv_hash, constants) |
filter_legacy_meta.py | meta-keyword filtering of legacy corpora (META_KEYWORDS on first human prompt) |
trim_long_rows.py | row token ceiling (drop >5100 tok rows for the OOM-safe corpus) |
build_armb2_corpus.py | arm-B2 assembly: 217 v28-glm + 60 todo_real + 72 todo_condensed + 87 restraint rows |
qwen38_train.py | QLoRA train (Unsloth; 214 steps; ckpts 175/200/214) |
merge_qwen38_pure.py | shard-streamed BF16 merge (MTP head preserved, vocab patched) |
build_todo_probes.py | probe battery build (T1–T3) |
build_no_drift_probes.py | tier-4 no-drift probes (10 off-topic asks) |
eval_todo_probes.py | battery runner (llama-server, tier-aware scoring incl. drift column) |
openai_translate_probe.py | BFCL drop-in check (Hermes XML → OpenAI tool_calls round-trip) |
The scripts carry the author's machine paths (F:/models/..., C:/AI/...)
and run on this rig's ROCm stack (Unsloth 2026.8.18, torch 2.11 ROCm,
RX 7700 XT). The todo_real/todo_condensed rows derive from DJLougen
Hermes traces (Apache-2.0); the restraint rows are real no-call asks from
the author's own agent sessions — training data is not published (same
policy as v26/v27). Zero author-session tool-call traces entered this
corpus except through the published DJLougen lineage.
Convert/quantize: llama-rocmfpx fork, convert_hf_to_gguf.py --outtype bf16 → llama-quantize Q4_0_ROCMFP4_FAST (see GGUF repo).
Quantized version
Q4_0_ROCMFP4_FAST GGUF (ROCmFPX, AMD RDNA3 kernels) is published in the
companion repo. ROCmFPX quants target AMD ROCm inference; for portable use,
convert from the BF16 merge here.
Acknowledgements
Base model and license inherited from empero-ai/Qwen3.8-9B-Distill (Apache-2.0). Teacher trajectories from GLM-5.3-Flash via the HuggingFace router; real traces from DJLougen Hermes traces (Apache-2.0).