ModernBERT-base for prefix-aware legal-guidance detection
This model detects whether legal guidance is already present in the cumulative user-turn prefix of a multi-turn LLM conversation. It was trained for research on early detection and routing of legal information needs. It is not a legal advice system and must not be used to determine whether a person has a valid legal claim.
Task formulation
One example is generated after each user turn. Prefixes before
first_guidance_user_turn_id are negative; the onset prefix and all later
prefixes are positive. Every prefix from a guidance-negative conversation is
negative. The final prefix matches the user-only full-conversation
serialization used in the comparison model.
All prefixes are retained. Each prefix is weighted by the inverse of its conversation's number of user turns, so every conversation has total training weight 1.0.
Data
- Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
- Requested revision:
main(latest at run time) - Resolved revision:
489b87c18e571e8c7e9a3b1c2463b79e50cef120 - Training: 1,632 conversations / 4,865 prefixes
- Validation: 290 conversations / 912 prefixes
- Positive prefixes: 2,042 train / 376 validation
The first-guidance-turn annotations are silver labels and were not independently human-validated for this run. Onset metrics must therefore not be described as gold evaluation.
Training configuration
| Setting | Value |
|---|---|
| Base model | answerdotai/ModernBERT-base |
| Resolved base revision | 8949b909ec900327062f0ebf497f51aef5e6f0c8 |
| GPU | NVIDIA A100-SXM4-40GB (39.5 GiB) |
| Maximum length | 4096 |
| Epochs | 3 |
| Learning rate | 5e-05 |
| Effective batch size | 32 |
| Weight decay | 0.01 |
| Seed | 42 |
| Loss weighting | Equal total weight per conversation |
| W&B run | https://wandb.ai/rl-research-team/legal-guidance-prefix-v3/runs/z751a3vj |
Validation results
The threshold 0.51 was selected on the silver validation split by
conversation-weighted prefix macro-F1, with positive-class F1 as tie-breaker.
| Evaluation | Macro-F1 | Positive F1 | AUPRC | Accuracy | Balanced accuracy |
|---|---|---|---|---|---|
| All prefixes, conversation-weighted | 0.8727 | 0.8530 | 0.9065 | 0.8758 | 0.8789 |
| All prefixes, unweighted | 0.8075 | 0.7840 | 0.8855 | 0.8103 | 0.8140 |
| Final/full prefixes | 0.8685 | 0.8603 | 0.9180 | 0.8690 | 0.8693 |
Silver onset analysis
| Measure | Value |
|---|---|
| Negative-conversation false-alarm rate | 0.1603 |
| Pre-onset false-alarm rate | 0.1269 |
| Missed-guidance rate | 0.0746 |
| Exact onset accuracy | 0.7612 |
| Within-one-user-turn accuracy | 0.8433 |
| Mean absolute turn error among detected positives | 0.4113 |
Repository artifacts
prefix_validation_metrics.json: complete metrics and threshold curveprefix_validation_predictions.csv: per-prefix probabilities and predictionsonset_validation_predictions.csv: conversation-level onset predictionsrun_metadata.json: model, data, software, and hardware provenance
Limitations
The dataset is English-language, jurisdiction-agnostic, and derived from public
LLM interaction logs that may contain sensitive content. The training and onset
labels are silver labels. The model was trained with one seed and has not yet
been evaluated on the adjudicated 200-conversation human gold set. Final-prefix
guidance evaluation on gold is appropriate after adjudication; onset evaluation
on gold requires separate human review of first_guidance_user_turn_id.