SchemaForge V2 — Distilled MiniCPM5-1B (V2 final, iteration 15)
This is the V2 release checkpoint. It closes out the V2 phase of the SchemaForge project: the pipeline (teacher-validation gate, hard-example generation, confidence calibration, hybrid routing, failure-category analysis, and a now-deterministic/reproducible training pipeline) is considered methodologically complete as of this checkpoint. Follow-on work (V3) targets the model's own remaining weaknesses (see Known limitations) under a fresh whitepaper. See SchemaForge V2 whitepaper and
logs/V2_TRAINING_FAILURES.mdfor the complete 15-iteration research history, including every negative result along the way.
What this is
A student checkpoint (openbmb/MiniCPM5-1B, ~1.04B params) sequence-level knowledge-distilled
from a google/gemma-4-31B teacher for JSON/structured extraction across 12 document schemas
(invoice, receipt, resume, contract, support ticket, medical note, insurance claim, CRM record,
email, conversation, form, knowledge-graph triple). Part of the SchemaForge V2 project: an
automatic distillation pipeline with teacher-output validation, hard-example generation, and
field-level evaluation, not just a checkpoint — and the pipeline itself is the project's primary
contribution. This specific checkpoint's headline result is not a raw-model number: it's that
routing between this model and a tuned deterministic pre-pass (by field ownership) beats either
system alone on every metric simultaneously — see Results below.
Why this checkpoint, specifically
15 training iterations were run during V2 development (full history in
logs/V2_TRAINING_FAILURES.md). The single highest field-F1 measurement across all of them
(0.6858, iteration 5/10) was produced by a checkpoint whose weights were later lost to a since-
fixed process bug (a checkpoint got overwritten without a backup before that safeguard existed —
documented, not hidden, in iteration 12's postmortem). This checkpoint (iteration 15) is the
most recent, and was produced by the fully bug-fixed pipeline: a fixed training random seed, an
automatic checkpoint-backup step, and — critically — deterministic (greedy) teacher-label
generation, none of which were true for the iteration-5 checkpoint. A controlled re-run
(iteration 15) measured this fix's actual impact: field F1 moved by only 0.0003 versus its
immediate predecessor, meaning this checkpoint's score (0.6827) is a stable, reproducible
measurement, not a lucky or unlucky draw — the property that matters most for a checkpoint meant
to be published and cited.
Training data
- 2700 hard-example records generated by
schemaforge/hardexamples/generate.pyacross the 9 training schemas (3 schemas —insurance_claim,conversation,kg_triple— are held out from training entirely, evaluated only for generalization). - Teacher outputs gated through a 4-step validation pipeline (JSON parse, Pydantic schema validation, source-support/ontology-derivation check, no-over-assertion check) before admission. 1625/2700 (60.2%) of teacher outputs passed the gate.
- 3 epochs, sequence-level cross-entropy distillation (no cross-tokenizer logit KL — found invalid
in an earlier project phase and dropped, see
docs/PROJECT_CHARTER.md§7.1). - Teacher generation uses greedy decoding (
temperature=0.0) for reproducibility — see "Why this checkpoint" above.
Results
On a 72-record held-out eval set spanning all 12 schemas (including the 3 held-out ones):
| system | field precision | field recall | field F1 | hallucination rate | schema validity |
|---|---|---|---|---|---|
| deterministic rules alone | 0.9185 | 0.1729 | 0.2911 | 0.0000 | 1.0000 |
| this model alone (residual-field prompt) | 0.4900 | 0.4784 | 0.4841 | 0.0914 | 0.8194 |
| hybrid (rules → this model) | 0.7174 | 0.6513 | 0.6827 | 0.0092 | 0.8333 |
The hybrid system beats both individual systems on every metric simultaneously — this is the
project's central architectural claim, and it holds here as it has in every hybrid configuration
tested. missing_field (omission) is the dominant failure category across every measured
configuration in this project (roughly 55-62% of all failures) — the clearest remaining target
for future work.
Known limitations
- Model-alone field F1 (0.4841) is close to, and on some measurements below, the un-distilled
base
MiniCPM5-1B's zero-shot score — the distillation's value in this pipeline is realized through the hybrid architecture, not as a standalone extractor. Use it hybrid, with the deterministic pre-pass, not alone. - Confidence output is poorly calibrated without post-hoc temperature scaling (raw mean-token confidence has been measured badly overconfident on this project's checkpoints); a fitted temperature scaler helps but was not re-fit specifically for this checkpoint.
- No hybrid escalation policy to a frontier LLM is implemented — the confidence signal has been characterized (coverage-at-risk curves exist) but no production routing threshold is set.
missing_fieldremains the dominant failure mode; a targeted fix attempted in iteration 14 (isolated corruption-operator training) did not help and is documented as a negative result.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
model = AutoModelForCausalLM.from_pretrained("<this-repo>", trust_remote_code=True)
prompt = (
"Extract the following fields as JSON from the text below. Schema: invoice.\n"
"Fields to extract: line_items[].description, vendor_name.\n"
"Text:\nINVOICE #INV-1001. Vendor: Acme Supply Co. ...\n"
"JSON Output:"
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
For production use, run this model behind the deterministic pre-pass (schemaforge/deterministic/)
and merge with schemaforge/hybrid/pipeline.py's merge_prediction — using the model alone
forgoes the architecture's actual advantage. The prompt format (schema name + explicit residual
field list, generated from the schema's semantic_fields minus whatever the deterministic pass
already resolved) must match training; see src/08_hybrid_eval.py's build_prompt in the source
repository for the exact template.
Training/compute
AMD Instinct MI300X (192GB), ROCm 7.0.2, PyTorch 2.11.0.dev+rocm7.0. GPU access provided by the AMD AI Developer Program.
Checkpoint identity
model.safetensorssha256:c13f7f6c21c35d2ed9159acc044a9f88abcd9fdfe9d0248ee625236b70c8470d- Training run:
loop-iter3-20260810T095747Z(seeexperiments/loop-iter3-20260810T095747Z/in the source repository for full machine-readable provenance) - Git commit:
9c941ea12f299b66c7bfd04603f6beea0ce04c9b
Source repository
See the SchemaForge GitHub repository for the full pipeline (schema registry, deterministic pre-pass, hard-example generator, teacher-validation gate, evaluation harness, calibration module, failure-category classifier, hybrid routing), the V2/V3 whitepapers, all experiment manifests, and evidence graphs.