roberta-base-squad2-cuad
Extractive QA model for legal clause extraction: deepset/roberta-base-squad2 fine-tuned on CUAD v1 (510 commercial contracts, 41 clause categories, annotated by lawyers).
This is the tier-2 span extractor behind Verity, a contract-review pipeline. A zero-shot classifier assigns a clause type; this model extracts the verbatim clause text with exact character offsets. Training and evaluation code is in the repo's cuad/ directory and reproduces this model end to end with uv run python cuad/run.py.
Results
Evaluated on a held-out test set of 51 contracts, all 41 CUAD categories.
| Model | Macro F1 (trimmed) | Macro F1 (full) | Macro EM |
|---|---|---|---|
| BART-MNLI zero-shot baseline | 0.41 | 0.66 | 0.66 |
| this model (base) | 0.73 | 0.83 | 0.81 |
| roberta-large variant | 0.75 | 0.84 | 0.82 |
Trimmed macro F1 excludes categories with fewer than 30 positive spans in the test set, where per-category F1 is unstable. The trimmed figure is the meaningful one for model comparison.
Split is a deterministic alphabetical contract-level 80/10/10, chosen for reproducibility. Results are not directly comparable to papers using a different split.
Usage
The model was fine-tuned on the verbatim CUAD category questions and expects them at inference. Using free-form questions degrades extraction quality.
from transformers import pipeline
qa = pipeline("question-answering", model="mjolnirslams/roberta-base-squad2-cuad")
question = (
'Highlight the parts (if any) of this contract related to "Governing Law" '
"that should be reviewed by a lawyer. Details: Which state/country's law "
"governs the interpretation of the contract?"
)
result = qa(question=question, context=contract_text)
# {'answer': 'This Agreement shall be governed by the laws of the State of New York.', 'start': ..., 'end': ..., 'score': ...}
The full 41-question mapping ships with the Verity repo in cuad_questions.json.
Like its SQuAD 2.0 base, the model predicts no-answer when the clause is absent from the context. Contexts longer than 512 tokens need sliding-window chunking (the question-answering pipeline handles this; Verity chunks contracts upstream).
Training
| Hyperparameter | Value |
|---|---|
| Base model | deepset/roberta-base-squad2 |
| Epochs | 2 |
| Batch size | 128 |
| Learning rate | 4e-5 |
| Warmup ratio | 0.1 |
| Weight decay | 0.01 |
| Max sequence length | 512 |
| Doc stride | 128 |
| Precision | bf16 |
Caveats
Corpus skew. CUAD is heavily skewed toward US commercial contracts. Performance on contracts from other jurisdictions, or contract types underrepresented in CUAD (employment agreements, consumer contracts), is unknown. Fine-tune further before deploying to those contexts.
Span contiguity. Extractive QA treats every answer as a contiguous span. CUAD occasionally annotates disjoint spans for a single (contract, category) pair; the model predicts one span per forward pass, which caps recall for those categories.
Question format. Extraction quality is tied to the verbatim CUAD questions above. This is not a general-purpose legal QA model.
License and attribution
- Base model: deepset/roberta-base-squad2 (CC BY 4.0)
- Data: CUAD v1, The Atticus Project (CC BY 4.0). Hendrycks, Burns, Chen, Ball. "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review." NeurIPS 2021.
- This model: CC BY 4.0