DistilBERT — Patent CPC Section Classifier
Fine-tuned distilbert-base-uncased that reads a patent abstract and predicts its CPC section — the top, broadest level of the Cooperative Patent Classification hierarchy (a single letter, A–H plus Y; 9 classes).
"Section", not the whole CPC code. CPC is hierarchical — section (
G, 9 classes) → class (G06, ~130) → subclass (G06F, ~660) → group/subgroup. This model predicts only the section (the first letter), which is the coarsest and easiest level.
Labels
| id | CPC | Section |
|---|---|---|
| 0 | A | Human Necessities |
| 1 | B | Performing Operations; Transporting |
| 2 | C | Chemistry; Metallurgy |
| 3 | D | Textiles; Paper |
| 4 | E | Fixed Constructions |
| 5 | F | Mechanical Engineering; Lighting; Heating; Weapons; Blasting |
| 6 | G | Physics |
| 7 | H | Electricity |
| 8 | Y | General tagging of new or cross-sectional technology |
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="crismolav/distilbert-patent-cpc-classifier", top_k=3)
clf("a steam turbine exhaust hood includes an exhaust hood section and a butterfly "
"plate arranged in the exhaust hood section with a complex curvilinear profile.")
# -> [{'label': 'Mechanical Engineering; ...', 'score': ...}, ...]
Results (held-out test split, 5,000 abstracts)
Macro-F1 is the headline metric because the classes are heavily imbalanced (~26× between the largest and smallest section), so plain accuracy is misleading.
| Model | Macro-F1 | Accuracy |
|---|---|---|
| TF-IDF + Logistic Regression (baseline) | 0.558 | 0.594 |
| DistilBERT (this model) | 0.613 | 0.667 |
In context — on the same dataset, the LSG Attention paper (Table 2) reports RoBERTa-512 at 0.618 macro-F1 / 0.666 micro-F1, and the best long-document model (LSG) at ~0.644 macro-F1. This model matches the RoBERTa benchmark using a smaller, distilled architecture on abstracts truncated to 256 tokens — i.e. it sits ~3 points below the dataset's practical ceiling.
Per-class F1
Strongest: Human Necessities (0.76), Electricity (0.75), Physics (0.72).
Weakest by far: CPC Y (0.11) — "general / cross-sectional technology". Y is applied on top of a primary technical area (e.g. a climate-friendly chemistry patent is tagged both C and Y), so as a single-label target it overlaps every other section and is intrinsically hard.
Top-k accuracy — the single-label ceiling is partly artificial
CPC is inherently multi-label (a patent carries ~1–3 codes on average and often spans several sections); this dataset keeps only the primary one. So many top-1 "errors" are the model naming a genuinely valid co-section rather than misreading the patent. On the test split:
| top-1 | top-2 | top-3 | |
|---|---|---|---|
| Overall | 0.667 | 0.843 | 0.927 |
Class Y | 0.072 | 0.732 | 0.978 |
Y is ranked #1 only ~7% of the time but sits in the top-2 ~73% / top-3 ~98% — the model recognises Y patents, it just ranks their concrete technical section (C, F, H…) above the cross-cutting Y tag. Much of the apparent error is the single-label reduction of a multi-label task, not the model being wrong.
Training data
ccdv/patent-classification, abstract config (derived from BIGPATENT). Official splits: 25k / 5k / 5k.
The training split was cleaned for integrity before fine-tuning:
- removed 192 exact-duplicate abstracts;
- removed 149 abstracts that leaked from train into the validation/test splits (which would otherwise inflate the test score);
- eval splits left untouched.
Net training set: 24,660 abstracts (the 340 removed rows are ~1.4% of train).
Training procedure
- Base:
distilbert-base-uncased - Max sequence length: 256 tokens (covers ~97% of abstracts; only ~3% are truncated)
- 3 epochs, learning rate 2e-5, batch size 16, best checkpoint selected by validation macro-F1
- Hardware: single Apple Silicon GPU (MPS)
Experiment: class-weighted loss
An inverse-frequency ("balanced") class-weighted cross-entropy was also tried to combat the imbalance. On test it was roughly a wash for macro-F1 (0.614 vs 0.613) at a small accuracy cost, but it more than doubled the worst class (Y F1 0.11 → 0.21, recall 0.07 → 0.18). The takeaway: inverse-frequency weighting redistributes performance toward rare classes rather than strictly improving the mean, and Y's difficulty is semantic overlap, not frequency. This published checkpoint is the unweighted model (higher accuracy, equal macro-F1).
Limitations & intended use
- Scope: English patent abstracts; predicts CPC section only (not class/subclass). Not validated on claims, full text, or non-English patents.
Ysection is unreliable (see above) — treat low-confidenceYpredictions with caution.- Research / portfolio model, not for production IP decisions. CPC assignment in practice is multi-label and done by trained examiners.
- Inherits biases of the underlying patent corpus (US-centric, time-bounded).
Citation
- Dataset & long-document benchmark: Condevaux & Harispe, LSG Attention (arXiv:2210.15497);
ccdv/patent-classification. - Base model: Sanh et al., DistilBERT (arXiv:1910.01108).