Applied NLP / NER — Stage 3: First Model Run
Fine-tuned distilbert-base-cased (65.2M params, ~250MB fp32 on disk) on
Ali-0111/applied-nlp-ner-stage2-expanded
(644 train / 115 test records) for 8-label NER token classification (BIO scheme, 17 tags).
Why this model (decision D1)
NER leans heavily on capitalization as a signal, so a cased model was preferred over uncased.
distilbert-base-cased is the smallest well-supported cased token-classification checkpoint with
a strong NER track record, over a genuinely-smaller-but-uncased-only alternative
(google/electra-small-discriminator, 14M) or an even tinier uncased option
(prajjwal1/bert-mini/bert-small, 11-29M) that would have sacrificed the capitalization signal
and risked quality on the dataset's thinnest labels (WORKOFART, PRODUCT).
Model stats
- Parameters: 65,203,985
- Size on disk (fp32 safetensors): ~250 MB
- CPU inference (4 threads, batch size 1, RTX-3060-laptop-class dev machine,
CUDA_VISIBLE_DEVICES=""):- RSS after model load: ~884 MB (includes PyTorch/Transformers runtime overhead, not just weights)
- RSS during inference: ~1.1 GB
- Throughput: ~35 samples/sec, ~1,780 tokens/sec (single-threaded batch-of-1; ~29ms/sample latency)
- Training hardware: RTX 3060 Laptop GPU (6GB VRAM), 6 epochs, batch size 16, lr 5e-5, ~23s total training time (644 records)
Training setup (reproducible)
- Base model:
distilbert-base-cased - Max sequence length: 512 (only 1 of 759 records exceeds this token count)
- Epochs: 6, batch size: 16, learning rate: 5e-5, weight decay: 0.01, seed: 42
- No validation split used (per the task brief's own guidance — not enough data/compute for a 3-way split at this scale; train/test is the recommended setup)
- Scripts:
stage3_model/scripts/prepare_bio.py(char-span to BIO alignment via tokenizer offset mapping),stage3_model/scripts/train.py(training + evaluation),stage3_model/scripts/measure_cpu_inference.py
Metrics
Train set
| Label | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| AMOUNT | 0.944 | 0.960 | 0.952 | 977 |
| JOB | 0.928 | 0.957 | 0.942 | 536 |
| LOCATION | 0.904 | 0.925 | 0.915 | 603 |
| ORGANIZATION | 0.933 | 0.957 | 0.945 | 717 |
| PERSON | 0.942 | 0.929 | 0.936 | 666 |
| PRODUCT | 0.815 | 0.837 | 0.826 | 190 |
| TIMEDATE | 0.931 | 0.931 | 0.931 | 825 |
| WORKOFART | 0.939 | 0.946 | 0.942 | 277 |
Overall: P=0.927, R=0.940, F1=0.934, Accuracy=0.986
Test set
| Label | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| AMOUNT | 0.915 | 0.950 | 0.932 | 260 |
| JOB | 0.723 | 0.754 | 0.738 | 114 |
| LOCATION | 0.723 | 0.802 | 0.760 | 91 |
| ORGANIZATION | 0.661 | 0.757 | 0.706 | 111 |
| PERSON | 0.825 | 0.819 | 0.822 | 138 |
| PRODUCT | 0.667 | 0.636 | 0.651 | 22 |
| TIMEDATE | 0.762 | 0.752 | 0.757 | 145 |
| WORKOFART | 0.719 | 0.767 | 0.742 | 60 |
Overall: P=0.786, R=0.820, F1=0.803, Accuracy=0.933
The train/test gap (F1 0.93 vs 0.80) reflects the small training set (644 records) relative to a 65M-param model over 6 epochs — expected at this scale, and exactly what Stage 4's diagnose/relabel/retrain step is meant to address.