Qwen3.8-9B Hermes FC — Tooluse
A QLoRA continue-tune of the v27 todo model
(qwen3.8-9b-hermes-fc-todo), trained on 218 high-verification
teacher-distilled trajectories generated by GLM-5.3-Flash. This is the
shipping checkpoint of the v28 milestone (checkpoint 175 of 214; the
final checkpoint overfits — see Evaluation).
Addendum 2026-09-03 — evaluation-methodology correction
Read this before you use the numbers below.
After this checkpoint was published, we tested the base model
(Empero/Qwen3.8-9B-Distill, no fine-tuning) through the live Hermes
interface: the OpenAI-native tools parameter path that Hermes actually
uses. The base model calls tools correctly on its own.
The harness that certified this checkpoint did not use that interface.
It baked tool schemas into the system text, never passed tools, and
required raw <tool_call> XML in the response. That prompt shape matches
neither the base model's own chat template nor the live Hermes request
path. Every gate score in the v26-to-v29 lineage measured this synthetic
interface.
The statement in this README that "the base distill contributes zero tool-calling" is incorrect. It measured the synthetic interface.
Corrected comparison (native tools path, same 4-tool battery, greedy):
| model | tier-1 fired | tier-1 name=todo | tier-2 todo-first | tier-3 fired (want 0) | tier-4 fired (want 0) |
|---|---|---|---|---|---|
| base (no fine-tune) | 18/20 | 17/20 | 3/10 | 5/10 | 0/10 |
| v28 ckpt-175 (this model) | 17/20 | 10/20 | 8/10 | 9/10 | 4/10 |
| armB2-175 (clean retrain) | 10/20 | 8/20 | 10/10 | 4/10 | 0/10 |
What the fine-tune lineage actually changed: todo-first planning on multi-step tasks (tier-2 name=todo 3/10 base to 8/10 v28 to 10/10 armB2). That is real, and it is the one capability the lineage added. The costs: restraint regressions (tier-3 over-trigger 5/10 to 9/10; tier-4 tool-drift on trivia 0/10 to 4/10) and, in armB2, over-suppression (tier-1 10/20).
Status: the fine-tune program is frozen as of 2026-09-03. The base model is the author's runtime default. This checkpoint stays published as the strongest measured demonstration of SFT-taught todo-first planning in the lineage, and as the artifact that exposed the harness error. The Evaluation section below describes behavior on the synthetic interface; do not read it as native-interface competence.
Reproduction: scripts/native_battery.py + scripts/todo_probes.jsonl
scripts/no_drift_probes.jsonlproduce the corrected table (any GGUF,CTX=65536to match the serve config).
- Base model: Empero/Qwen3.8-9B (Apache-2.0)
- Warm start: v27 merged weights (
qwen3.8-9b-hermes-fc-todo), QLoRA adapter on top (envelope + tool tokens already baked in) - Training data: 218 trajectories (ShareGPT format, 551 todo calls),
generated by GLM-5.3-Flash via the HuggingFace router on 112 synthetic
tasks mined from the author's agent corpus — not the author's own traces:
- 190 teacher trajectories (85% verified by sandbox execution; failures with recovery injected for 15.8% of rows)
- 61 rows carry distilled short-chain-of-thought; 159 envelope-only
- 12 synthetic web-search tasks (first web signal)
- Full dataset published separately (see
hermes-fc-v28-data); probes were kept virgin (zero overlap with generation or training)
- Method: QLoRA 4-bit, rank 16 / alpha 16, dropout 0, batch 1 x grad-accum 8 (effective 8), lr 2e-4, warmup 0.1, MAX_SEQ 6144, 8 epochs = 214 steps (final loss 0.02085, avg train loss 0.09405)
- Vocabulary: 248077 → 248079 (2 added tool tokens:
<|tool_call|>,<|tool_response|>) - Weights: BF16 full merge (13 shards, 18.9 GB, 9.20B params) plus a ROCmFPX quantized GGUF (see companion repo)
- Architecture note: merge preserves the 15-key MTP (multi-token
prediction) head from the base —
Qwen3_5ForConditionalGenerationwithtext_config.vocab_size = 248079. Visual tower dropped (text-only).
Evaluation (todo-probe battery, greedy, todo schema in every prompt)
40 probes, 3 tiers: Tier 1 = 20 real DJLougen todo trajectories held out of ALL training generations (hash-exact, zero leakage; GT = first todo call). Tier 2 = 10 synthetic multi-step tasks (refactor+tests+docs, CI setup, release prep) → autonomous todo reach-for. Tier 3 = 10 trivial asks (want 0/10) → over-trigger guard.
| tier | model | fired | name=todo | todos_ok | format_exact |
|---|---|---|---|---|---|
| T1 (real, held-out) | v27 | 13/20 | 11/20 | 11/20 | 0/20 |
| T1 | v28 ckpt-175 (this model) | 17/20 | 16/20 | 16/20 | 4/20 |
| T2 (novel multi-step) | v27 | 9/10 | 7/10 | 7/10 | 3/10 |
| T2 | v28 ckpt-175 | 8/10 | 7/10 | 7/10 | 4/10 |
| T3 (trivial, want 0) | v27 | 5/10 | 1/10 | 1/10 | 2/10 |
| T3 | v28 ckpt-175 | 5/10 | 1/10 | 1/10 | 4/10 |
Headline: tier-1 held-out fidelity 13→17 fired / 11→16 todos_ok — the strongest held-out result in the v2→v28 lineage. Tier-2 holds at 7/10. The gate is reproducible: run twice (pre-flip and post-flip), identical scores.
Why ckpt-175 and not ckpt-214: the final checkpoint (214) regressed on the tier-3 over-trigger guard (8/10 fired, 6/10 unconditional todo reaches on trivial asks). Root cause: the v28 corpus has zero no-call / just-answer rows, so the model trained past the point where it can decline a tool call. ckpt-175 at epoch ~6.5 is the sweet spot: full tier-1/2 capability, tier-3 still within the v27 bar. This is the "more data ≠ better" lesson, measured in-checkpoint this time.
format_exact caveat (important): the format column demands
envelope-only output (^\s*<tool_call>.*?</tool_call>\s*$). v28's low
format_exact is the same phenomenon documented in v27: perfect
envelopes but entire multi-turn agentic loops in one completion
(<tool_call>todo</tool_call> → simulated user <tool_response> →
next <tool_call>search_files</tool_call> → …). The live Hermes runtime
parses this fine (verified in real sessions); it just fails the
envelope-only regex.
Known limitation (no-call / memory reflex): because the corpus has no
no-call rows, the model will still reach for a tool on questions that
need none (e.g. a trivia ask triggers a tool call rather than a direct
answer). It also reaches for tools it was never trained on when the
runtime exposes them (observed: memory calls with invented facts on a
trivia question; the runtime guardrails rejected them). Use within an
agent harness with validation; do not use for open-ended chat without
tool-scope enforcement.
The base distill's tool-calling was measured at zero on the synthetic harness only; through the native interface it fires 18/20 (see the Addendum at the top of this card).
Intended use
Research artifact for experimenters working on tool-call behavior in
9B-class models — not a product. The tool_call XML envelope and tool
schemas match the Hermes agent runtime this was trained on. This is an
intermediate milestone in an iterative recipe series (v1 → v2 → v25 →
v26 → v27 → v28); v29 is in design (RAG via docs-index + clarify, plus a
clean-base vs warm-lineage A/B).
What this repo contains
- The BF16 merged weights (this repo)
- All scripts that produced the model and dataset (
scripts/)
Reproducibility
All scripts that produced this model are in scripts/:
| file | purpose | number it produced |
|---|---|---|
build_v26.py | base dataset build (imported for build_system, pad_to_bucket, conv_hash, constants) | shared helpers |
mine_v28_tasks.py | task mining from DJLougen corpus (Jaccard-dedup vs probes + v27 training set) | 112 tasks |
analyze_v28_tasks.py | task-prompt sanity analysis (dedup, leakage, verify-kind) | task stats |
glm_harness_v28.py | GLM-5.3-Flash trajectory generator: router calls, 6-tool schema, hermetic sandbox, verify() filtering, retry/backoff | 224 raw → 190 verified trajectories |
inject_failures_v28.py | failure injection (1–2 format-correct failures/trajectory, all with recovery) | 30 injected rows |
split_reasoning_v28.py | reasoning split: envelope-only vs distilled short-CoT bands | 159 / 61 rows |
assemble_v28.py | final assembly → train_v28.jsonl (218 rows, 551 todo calls, gate: 0 no-call, 2 too-long) | row count, mix |
qwen38_train_v28.py | QLoRA continue-train via Unsloth (8 epochs, 214 steps, loss 0.02085) | training loss/steps |
merge_qwen38_v28_pure.py | shard-streamed BF16 merge, final checkpoint (MTP head preserved; vocab patched) | 13 shards / 18.9 GB |
merge_qwen38_v28_175_pure.py | same, checkpoint 175 (the shipped checkpoint) | 13 shards / 18.9 GB |
build_todo_probes.py | 40-probe battery build (T1 held-out / T2 multi-step / T3 trivial) | the table above |
eval_todo_probes.py | probe battery runner (llama-server on 9088, tier-aware scoring) | the table above |
native_battery.py + todo_probes.jsonl + no_drift_probes.jsonl | native-interface battery (API tools param, live-Hermes path) | the Addendum table (2026-09-03) |
eval_fc_heldout.py + convert_hf_hermes.py | 45-probe held-out eval + HF-conversion helper (kill-servers pattern source) | legacy eval, import support |
The scripts carry the author's machine paths (F:/models/..., C:/AI/...)
and run on this rig's ROCm stack (Unsloth 2026.8.18, torch 2.11 ROCm,
RX 7700 XT); adjust paths for your environment. The sandbox used for
generation is hermetic and wiped per run — no real user data entered the
training trajectories.
Convert/quantize: llama-rocmfpx fork, convert_hf_to_gguf.py --outtype bf16
→ llama-quantize Q4_0_ROCMFP4_FAST (see GGUF repo).
Quantized version
Q4_0_ROCMFP4_FAST GGUF (ROCmFPX format, AMD RDNA3 kernels) is published in
the companion repo. ROCmFPX quants target AMD ROCm inference; for portable
use, convert from the BF16 merge here.
Training data
Published separately as hermes-fc-v28-data (Apache-2.0): train_v28.jsonl
(218 rows), tasks_v28.jsonl (112 tasks), and the harness/build scripts.
Teacher: GLM-5.3-Flash via the HuggingFace router; task corpus derived from
DJLougen Hermes traces (Apache-2.0). The v28 dataset contains zero traces
from the author's own sessions (hermetic sandboxes, synthetic tasks) — which
is what makes it publishable where v26/v27 were not.
Acknowledgements
Base model and license inherited from Empero/Qwen3.8-9B (Apache-2.0). Teacher distillation via GLM-5.3-Flash (Zhipu AI) through the HuggingFace router; task corpus derived from DJLougen Hermes traces (Apache-2.0).