Qwen3-4B CUAD Contract Clause Extractor

Qwen3-4B-Instruct fine-tuned with QLoRA to extract all 41 CUAD clause types from a contract in one pass, quoting the contract text verbatim and returning explicit empty lists for absent clauses. Runs in ~2.5 GB (q4_K_M GGUF) on consumer hardware.

Trained and evaluated on the official CUAD split: 408 training / 102 held-out test contracts, no contract overlap.

Training and evaluation code, raw eval outputs, and build notes: github.com/Ihtesham-star/cuad_llm_finetuning

Results

Detection F1: did the model find each clause type present in a contract (and produce a verbatim quote for it)? Scored per (contract, clause type) over 102 test contracts × 41 clause types; predictions merged across contract chunks. Bootstrap 95% CIs over contracts (n=1000).

Three seeds (42/43/44) under identical config: F1 0.8995 ± 0.0017. The published weights are seed 42.

Strict full-span coverage (a true positive requires the predicted quotes to fully contain every gold span for the clause — our reimplementation of the rule used by ContractEval, arXiv:2508.03080, which benchmarked zero-shot LLMs on this same 102-contract test set):

Caveats for that comparison: our matching is a reimplementation of their stated rule (whitespace-normalized, casefolded substring), not their harness; and ContractEval queries one clause type per call with its definition in the prompt, while this model answers all 41 clause types in a single call (~40× fewer calls per contract).

Data curve (seed 42, detection F1 vs number of training contracts): 50 → 0.746 · 150 → 0.862 · 300 → 0.885 · 408 → 0.900. The zero-shot 14B baseline (0.816) is passed somewhere between 50 and 150 annotated contracts.

Serving benchmark and comparison with Claude Opus 5.5

How the model behaves in production, measured in cuad-serving-bench (same 150-chunk CUAD test set, one RTX 5090, load tests up to 32 simultaneous users, cost per 1,000 contracts):

• The fine-tuned model is more accurate than Claude Opus 5.5 zero-shot (paired bootstrap, BF16 vs Claude: -0.031 F1, 95% CI -0.048 to -0.014) at roughly 1/50 of the cost.
• FP8 (llm-compressor) keeps the accuracy of 16-bit with 45% less memory; the Q4_K_M GGUF in this repo also keeps full accuracy (F1 0.900). AWQ 4-bit loses 1.5 F1 points.
• For many simultaneous users, serve it with vLLM: at 16 users vLLM FP8 does 2.2x the throughput of Ollama. For a single user on a desktop, the Q4_K_M GGUF with Ollama is the fastest option.
• A one-command private deployment (vLLM + FastAPI, Docker Compose) is in the benchmark repo under serve/.

Prompt format

The model was trained on exactly one prompt shape. Use it verbatim; other phrasings are out of distribution.

System:

You are a contract review system. Extract the requested clause types from the contract excerpt. Quote the contract text verbatim. A clause type not present in this excerpt gets an empty list. Reply with JSON only.

User:

Extract all 41 CUAD clause types from this contract excerpt as JSON (keys: clause types, values: lists of verbatim quotes).

CONTRACT EXCERPT:
<up to ~60k characters of contract text>

Output: one JSON object with all 41 CUAD clause names as keys, each a list of verbatim quotes (empty list = clause absent). Contracts longer than ~60k characters should be chunked with overlap and the per-chunk outputs merged (union per clause).

Usage

Ollama (recommended for local use)

ollama create cuad-extractor -f Modelfile   # FROM ./cuad-4b-s42-q4_K_M.gguf

Call with format: "json", temperature: 0, and num_ctx sized to fit your chunk plus the output (silent prompt truncation is the classic failure mode). On a small fraction of dense chunks (~6% in our runs) greedy decoding enters a repetition loop and hits the output cap; retrying that chunk with repeat_penalty: 1.05 (still temperature 0, deterministic) recovered nearly all cases in our evaluation, escalating to 1.15/1.25 for the rare holdouts.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor",
                                             torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor")

Repository layout: fp16 merged weights at the root (from_pretrained works directly), gguf/ holds the q4_K_M quantization measured above, lora/ holds the LoRA adapter (r=16) if you prefer to apply it to the base model yourself.

Training

• Base: unsloth/Qwen3-4B-Instruct-2507, QLoRA (4-bit base, LoRA r=16, alpha=32, all attention + MLP projections)
• 619 chunk-level examples from the 408 official training contracts (one example whose rendered length exceeded the 24,576-token training window was excluded and recorded)
• 2 epochs, lr 2e-4 linear with 6% warmup, effective batch 8, ~45 min on one RTX 5090
• Target: the gold JSON only (train_on_responses_only); empty lists for absent clauses are part of the target, which is what teaches calibrated "not present" behavior

Limitations

• Strict-coverage recall is the weak spot (0.58): when a clause is stated in several places, the model often quotes the main span but not every occurrence.
• Greedy decoding can loop on dense chunks (see the fallback note above).
• Trained on CUAD: English-language commercial contracts from SEC EDGAR filings. Performance on other contract styles, other languages, or non-contract legal text is unmeasured.
• This is a research/engineering artifact, not legal advice; extractions need human review in any real workflow.

Attribution

• Dataset: CUAD (Contract Understanding Atticus Dataset), The Atticus Project, CC BY 4.0
• Base model: Qwen3-4B-Instruct-2507 (Apache 2.0), 4-bit variant by Unsloth
• Trained with Unsloth