applied-nlp-ner-stage3-distilbert
Stage 3 output of an applied-NLP NER project: distilbert-base-cased fine-tuned
for token classification on
applied-nlp-ner-stage2-expanded-798
(679 train / 119 test records), predicting the 8-label schema PERSON,
ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB, AMOUNT via a
standard BIO tag scheme (17 tags total).
Why this model
distilbert-base-cased (65M params) was chosen as the smallest reasonable
encoder for this task: it keeps case information, which is one of the
strongest signals for PERSON/ORGANIZATION/LOCATION recognition (an uncased
model, or a much smaller model like BERT-mini, would throw that away).
DistilBERT halves BERT-base's parameter count and layer count via
knowledge distillation while retaining most of its token-level performance,
which matters here since training and inference both ran on CPU.
Size / resource footprint
| Parameters | 65,203,985 |
| Checkpoint size on disk | 249.4 MB |
| Training time (CPU, 8 epochs, 679 records) | 1384 s (~23.1 min) |
| Inference throughput (CPU, batch 16) | 19.8 records/sec |
No GPU was available in the training environment; all numbers above are CPU-only.
Results on the Stage 2 test split (119 records)
Overall: precision 0.658, recall 0.717, F1 0.686, token accuracy 0.945.
| Label | Precision | Recall | F1 |
|---|---|---|---|
| PERSON | 0.871 | 0.885 | 0.878 |
| ORGANIZATION | 0.577 | 0.619 | 0.597 |
| LOCATION | 0.598 | 0.792 | 0.682 |
| TIMEDATE | 0.651 | 0.726 | 0.686 |
| PRODUCT | 0.375 | 0.600 | 0.462 |
| WORKOFART | 0.388 | 0.491 | 0.433 |
| JOB | 0.714 | 0.667 | 0.690 |
| AMOUNT | 0.761 | 0.700 | 0.729 |
PERSON is the strongest label (large, easy-signal training class). PRODUCT and WORKOFART are the weakest, which tracks their size in the training set (138 and 295 spans respectively, versus 351-666 for every other label) — see the "Known limitation" note in the Stage 2 dataset card.
QA on out-of-distribution samples
The model was also run on 20 hand-written sentences that appear nowhere in
Stage 1 or Stage 2 (see data/stage3/wild_samples.json /
data/stage3/wild_predictions.json in the project repo), to see how it
behaves outside the register mix of Wikipedia/Wikinews/Gutenberg/bill-text.
Ten concrete, reproducible problem patterns came out of that pass; the full
list with examples is in the project report (report/REPORT.md, Stage 3
section). In short: multi-token span truncation (entities cut off after the
first token), a handful of spurious single-stopword spans ("The", "the"
tagged alone), and label confusion between ORGANIZATION/LOCATION/PRODUCT for
proper nouns whose surface form is ambiguous out of context (e.g. a sports
club sharing its name with a city) were the dominant failure modes -- not
outright missed entities.
Usage
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
tok = AutoTokenizer.from_pretrained("distilbert-base-cased") # placeholder, use this repo id
model = AutoModelForTokenClassification.from_pretrained("MatinAsgarov/applied-nlp-ner-stage3-distilbert")
tok = AutoTokenizer.from_pretrained("MatinAsgarov/applied-nlp-ner-stage3-distilbert")
nlp = pipeline("ner", model=model, tokenizer=tok, aggregation_strategy="simple")
nlp("Elon Musk announced that Tesla will open a factory in Shanghai.")