applied-nlp-ner-stage4-distilbert
Stage 4 output of an applied-NLP NER project: a retrain of
applied-nlp-ner-stage3-distilbert
after diagnosing its errors against 20 out-of-distribution "wild" sentences.
Same data as Stage 3 (Stage 2's train/test split — no relabeling was needed;
an audit of the training data found no mislabeled PRODUCT/WORKOFART spans to
fix) and the same architecture (distilbert-base-cased, 65.2M params).
Diagnosis carried over from Stage 3
PRODUCT (138 train spans) and WORKOFART (295 train spans) were 2-5x smaller than every other label (351-666 spans), and that data-scarcity, not a labeling error, explained most of the observed failure patterns.
Fix
- Oversampling — training records containing a PRODUCT or WORKOFART span duplicated ~2x (679 -> 902 records).
- More epochs — 8 -> 12, still selecting the best-eval checkpoint
(
load_best_model_at_endon F1), so extra training can only help.
Boundary truncation and spurious single-stopword spans were deliberately not patched with inference-time heuristics; see the project report for why.
Results: Stage 3 -> Stage 4 (same 119-record test split)
Overall: precision 0.658 -> 0.681, recall 0.717 -> 0.757, F1 0.686 -> 0.717.
| Label | Stage 3 F1 | Stage 4 F1 | Delta |
|---|---|---|---|
| PERSON | 0.878 | 0.912 | +0.034 |
| ORGANIZATION | 0.597 | 0.626 | +0.029 |
| LOCATION | 0.682 | 0.729 | +0.048 |
| TIMEDATE | 0.686 | 0.739 | +0.052 |
| PRODUCT | 0.462 | 0.462 | +0.000 |
| WORKOFART | 0.433 | 0.491 | +0.058 |
| JOB | 0.690 | 0.657 | -0.032 |
| AMOUNT | 0.729 | 0.774 | +0.044 |
Wild-sample re-check
The same 20 out-of-distribution sentences from Stage 3 were re-run through
this model (data/stage4/wild_predictions.json in the project repo). Full
pattern-by-pattern comparison is in report/REPORT.md. Summary: 3 of the 10
Stage 3 patterns were resolved outright (Samsung/NASA organization
mislabeling, football-score boundary truncation), several partially improved
(multi-word span merging is now more reliable — "Treaty of Versailles" and
"Inflation Reduction Act" are captured whole where they weren't consistently
before), 2 remain unresolved (Elon Musk / Liverpool disambiguation), and one
new regression appeared (Beethoven mistagged WORKOFART instead of PERSON) —
reported honestly rather than cherry-picked.
Size / resource footprint (CPU-only)
| Parameters | 65,203,985 |
| Checkpoint size on disk | 249.4 MB |
| Training time (12 epochs, 902 oversampled records) | 3336 s (~55.6 min) |
Usage
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
model = AutoModelForTokenClassification.from_pretrained("MatinAsgarov/applied-nlp-ner-stage4-distilbert")
tok = AutoTokenizer.from_pretrained("MatinAsgarov/applied-nlp-ner-stage4-distilbert")
nlp = pipeline("ner", model=model, tokenizer=tok, aggregation_strategy="simple")
nlp("Samsung unveiled the Galaxy S25 Ultra in San Jose.")