mailroom-modernbert-classifier

Hierarchical document classifier for the LLM-Mailroom intake pipeline: a fine-tuned ModernBERT-base encoder with a doc_type head plus one subclass head per document class. It is the deterministic pre-check in the BERT-coupled intake overhaul (mailroom-issues #85).

Recommended production revision (M9a primary, Arm B): m9a-local-20260927-014429 — 3 epochs, loss_lambda_dt=0.65, held-out doc_type accuracy 0.9505, trainer validation gate_met: true. M9a mailroom-issues #112 subclass gates remain not met for contract / correspondence; use LLM sorter for subclass-ambiguous tails.

Hub releases (M9a, 2026-09-27)

Held-out test set: 323 documents, 8,192-token windows, never used for training, calibration fit, or threshold tuning. Authoritative metrics come from the eval_report_<tag>.json artifacts (not inline summary.json test fields when they differ).

Contractor-repo report mirrors (same content as sections below): M9a-REPORT-m9a-local-20260927-014429, M9a-REPORT-m9a-local-gpu1-armA-1ep-20260927-025236, M9a-COMPARE-ArmA-vs-ArmB-20260927.

Architecture

• Backbone: answerdotai/ModernBERT-base — 22 layers, 768 hidden, 8,192-token context, bf16.
• Heads: one head per class — doc_type (6 classes) plus 5 subclass heads (contract, corporate_record, correspondence, insurance_claim, merger_agreement). MLP heads with dropout 0.1.
• Windowing: token-level, 8,192 tokens with 512-token overlap. doc_type by plurality vote over windows; subclass by plurality over windows whose doc_type vote is the winning class.
• Calibration: per-head temperature scaling (temperatures.json).

Files

Labels

doc_type (6): contract, merger_agreement, corporate_record, correspondence, insurance_claim, unknown (inference-only abstention — not a trained class)

contract (24): agency, co_branding, collaboration, consulting, development, distributor, endorsement, franchise, hosting, ip, joint_venture, license, maintenance, manufacturing, marketing, other, outsourcing, promotion, reseller, service, sponsorship, strategic_alliance, supply, transportation

corporate_record (10): articles_of_incorporation, board_resolution, bylaws, charter_amendment, indenture, officer_certificate, other, powers_of_attorney, rights_instrument, subsidiary_list

correspondence (7): demand, email, letter, meeting_request, memo, notice, press_release

insurance_claim (6): auto, carrier, inpatient, outpatient, pde, property

merger_agreement (5): all_cash, all_stock, mixed_cash_stock, mixed_cash_stock_election, other

Usage

The backbone is a standard ModernBertModel; the hierarchical heads are a custom bundle. Load a pinned release tag (never a floating main tip in production):

from huggingface_hub import snapshot_download
from mailroom_ml.inference import load_bundle, classify_document

model_dir = snapshot_download(
    "Lucius-Morningstar/mailroom-modernbert-classifier",
    revision="m9a-local-20260927-014429",
)
bundle = load_bundle(model_dir)
result = classify_document(
    title="Notice of Default",
    text=document_text,
    bundle=bundle,
)
# result["doc_type"], result["subclass"], result["confidence"], result["route"]

route == "fast_path" means the calibrated gate passed and the LLM sorter may be skipped (skip mode + allowlist only); otherwise route the document to the LLM sorter with the BERT triage as an advisory prior.

Limitations

• Subclass heads are weak for contract / correspondence / corporate_record / merger_agreement on held-out test (M9a Arm B macro-F1 0.02–0.25). Use the doc_type head for routing; route subclass-ambiguous documents to the LLM sorter. Do not trust skip-mode subclass for these classes until #112 gates pass.
• Trained on a curated legal-document corpus; not a substitute for legal review.
• unknown is an inference-only abstention label, not a trained class.
• Long documents are windowed (8,192 tokens, 512 overlap); oversize documents fall back to the LLM path when routing policy requires it.

License

Apache-2.0 (inherits answerdotai/ModernBERT-base).

M9a full run report (2026-09-27)

The sections below merge the contractor-repo M9a reports published after local GPU training. Arm B is the production candidate; Arm A is a one-epoch λ ablation smoke on GPU 1.

M9a local Arm B — full report

Run tag: m9a-local-20260927-014429
Run ID: 20260927-064513
Arm: B (primary, GPU 0)
Checkpoint: data/modernbert_training/runs/m9a-local-20260927-014429/latest
Training log: logs/m9a-local-20260927-014429.log
Held-out eval JSON: reports/eval_m9a-local-20260927-014429.json → Hub eval_report_m9a-local-20260927-014429.json

Config

Validation (per epoch, observed doc_type macro-F1)

Trainer checkpoint selection (validation)

• Selected epoch: 3
• Selection gate_met: true (calibrated doc_type ECE ≤ 0.05, no doc_type macro-F1 regression beyond 0.005)
• Val doc_type macro-F1 (observed): 0.9389
• Val doc_type ECE (calibrated): 0.02
• Val subclass objective: 0.3334

Head exclusion policy (ECE > 0.05 → fast-path excluded)

Held-out test (323 docs @ 8,192 tokens)

Cohorts (#104 single vs multi-window)

Selective-risk sweep (global)

• budget_met: true
• recommended deployment threshold: 0.90

Per-head test macro-F1 (observed)

M9a #112 gates (held-out test)

Overall #112 gate verdict: FAIL (2/4 gates not met — contract and correspondence subclass heads remain below Verify-grade thresholds; doc_type and calibration gates pass).

Arm B verdict

• Doc-type routing: strong improvement vs run-3 baseline (0.8947 → 0.9505 test acc; ECE 0.0259).
• Trainer selection: epoch 3 gate_met true on validation.
• Subclass / #112: contract + correspondence test macro-F1 still block full #112 PASS; exclusion policy keeps high-ECE subclass heads off the fast path.
• Recommended production pointer: this run (Arm B, λ_dt=0.65) over the 1-epoch Arm A smoke.

M9a GPU-1 Arm A smoke — full report

Run tag: m9a-local-gpu1-armA-1ep-20260927-025236
Run ID: 20260927-075323
Arm: A (one-time GPU 1 smoke per logs/.m9a-gpu-policy)
Checkpoint: data/modernbert_training/runs/m9a-local-gpu1-armA-1ep-20260927-025236/latest
Training log: logs/m9a-local-gpu1-armA-1ep-20260927-025236.log
Held-out eval JSON: reports/eval_m9a-local-gpu1-armA-1ep-20260927-025236.json → Hub eval_report_m9a-local-gpu1-armA-1ep-20260927-025236.json
Published with --force-publish (smoke compare; gates not required).

Config

Validation (epoch 1)

No checkpoint met the lexicographic selection rule (calibrated doc_type ECE 0.0524 > 0.05 budget on the sole epoch). Trainer reports gate_met false and selection epoch 0 (placeholder).

Held-out test (323 docs @ 8,192 tokens)

Note: inline trainer test_metrics.doc_type_acc (0.8669) reflects a lighter in-train test pass; the authoritative held-out harness is the eval JSON above.

Cohorts

Selective-risk sweep refused (no per-head ECE sidecar in artifact for this 1-epoch smoke).

Per-head test macro-F1 (observed)

M9a #112 gates (held-out test)

Overall #112 gate verdict: FAIL (expected for 1-epoch λ_dt=0.5 compare arm).

Arm A verdict

Arm A smoke confirms λ_dt=0.5 + 1 epoch under-trains doc_type vs Arm B: −8.7 pp test doc_type acc and −10 pp conditional subclass acc vs the 3-epoch primary. Use for ablation only; do not promote to production default.

M9a Arm A vs Arm B — comparison

Held-out test (n=323)

Validation selection

M9a #112 gates (held-out test)

Interpretation

1. λ_dt and depth: Raising doc-type loss weight to 0.65 and training 3 epochs materially improves both routing accuracy and conditional subclass accuracy on the held-out set.
2. Production vs smoke: Arm B is the candidate release; Arm A documents the cost of under-training for gate and λ ablation.
3. Remaining gap: Both arms fail contract/correspondence subclass macro-F1 gates; Arm B additionally passes doc_type accuracy and calibration gates that Arm A misses.
4. Hub: Both checkpoints are tagged on this repo with distinct release tags and bundled eval_report_<tag>.json artifacts.

Reference baselines (doc_type test accuracy)

Historical — run-3 (2026-09-21)

Checkpoint runs/20260921-132753 (run_id 20260921-093211). Config: epochs=2, batch=4, grad-accum=8, lr=2e-5, loss_lambda_dt=0.65, sqrt-inverse weights, MLP heads. Held-out test doc_type accuracy 0.8947, conditional subclass 0.526, window ECE 0.0203. Superseded for routing by M9a Arm B (0.9505 doc_type acc).

Release 20261006-021245 (2026-10-06)

Held-out test eval artifact on this repo: eval_report_20261006-021245.json.

M9a #112 gates (held-out test)

Test surfaces

• subclass_accuracy_conditional: 0.6287
• window_calibration: ece=0.0152 band_ece=0.0682

Per-head test macro-F1 (observed)

Trainer selection (validation)

• selected epoch: 4
• gate_met: True
• subclass_objective: 0.42691999999999997
• doc_type macro-F1 (val): 0.9399
• doc_type ECE (val, calibrated): 0.0175
• head_exclusion_policy (ECE > 0.05):
  • contract: excluded=True ece=0.059
  • corporate_record: excluded=True ece=0.1391
  • correspondence: excluded=True ece=0.0745
  • doc_type: excluded=False ece=0.0175
  • insurance_claim: excluded=False ece=0.0
  • merger_agreement: excluded=True ece=0.2134