Polygraf / NER Stage 3: First Model
google/electra-small-discriminator fine-tuned end to end for token
classification on
inovruzova/polygraf-ner-stage2-expanded.
All layers were updated, not only the classification head.
This is the Stage 3 output of the Polygraf Applied NLP / NER technical task. It is a first model, published with its faults documented rather than tuned away.
Labels
Eight entity types, seventeen tags including O.
PERSON, ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB,
AMOUNT
The full annotation policy the labels follow is on the dataset card.
Usage
from transformers import pipeline
ner = pipeline(
"token-classification",
model="inovruzova/polygraf-ner-stage3-model",
aggregation_strategy="simple",
)
ner("The software engineer at Google filed the patent in March 2024.")
Model facts
| Base model | google/electra-small-discriminator |
| Parameters | 13,487,377 |
| Size on disk | 52.2 MB |
| CPU memory at inference | [MEASURE] MB |
| Throughput on CPU | 16.3 records per second, 61.5 ms per record |
Throughput measured one passage at a time on CPU, after warm-up, over 100 passages from the test split.
Training
| Hardware | NVIDIA RTX 4070 Laptop GPU, 8 GB VRAM |
| Learning rate | 1e-4 |
| Batch size | 8 |
| Epochs | 30 |
| Weight decay | 0.01 |
| Maximum sequence length | 512 |
| Seed | 202608 |
| Runtime | 131 seconds |
No validation split, following the task instruction, and therefore no checkpoint selection.
An earlier run used the Hugging Face defaults of 5e-5, batch 16 and 10 epochs and reached test F1 0.491. It was undertrained rather than overfitted: train F1 was 0.583 and the loss was still falling at the last step. The settings above were chosen from that evidence, all of which comes from the training split. 5e-5 suits BERT base at 110M parameters; this model is 13.5M.
Results
Micro-averaged F1: 0.745 on test, 0.999 on train.
Test, per label
| Label | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| TIMEDATE | 0.831 | 0.920 | 0.873 | 112 |
| PERSON | 0.805 | 0.901 | 0.850 | 101 |
| AMOUNT | 0.809 | 0.821 | 0.815 | 67 |
| LOCATION | 0.783 | 0.837 | 0.809 | 86 |
| JOB | 0.719 | 0.708 | 0.713 | 65 |
| PRODUCT | 0.578 | 0.673 | 0.622 | 55 |
| ORGANIZATION | 0.500 | 0.600 | 0.545 | 65 |
| WORKOFART | 0.480 | 0.490 | 0.485 | 49 |
| micro avg | 0.715 | 0.778 | 0.745 | 600 |
On train, every label scores between 0.997 and 1.000.
Limitations
The model memorises its training set. Train 0.999 against test 0.745 is a gap of 0.254. This is left uncorrected rather than fixed by shortening training, because with no validation split, shortening until the test gap narrows would be selecting a training length on test performance.
Three labels are much weaker than the rest. WORKOFART, ORGANIZATION and PRODUCT are exactly the labels governed by the four policy rules that need knowledge outside the token sequence: whether a spec-like token belongs to a product's commercial name, whether a brand is acting as a company, whether a title refers to a game or a film, and whether a place name stands for the institution it houses. Scarcity does not explain it; ORGANIZATION is the fourth most common label in the dataset.
The model is uncased. It never sees capitalisation, which is the strongest surface clue that a word is a name in English. Every model below about 35M parameters is uncased and the smallest cased one is five times larger, so this is the cost of choosing the smallest reasonable model.
It over-predicts. Recall exceeds precision on seven of eight labels.
Test metrics are agreement, not accuracy. The test labels were themselves measured at 98.3 percent span precision in a 40-record hand audit, so these numbers compare the model against an imperfect reference.
Two entities in the training data cannot be represented as token tags, because
the policy places a currency symbol outside the span and BERT-family tokenizers
split $ from a following digit but not £.
Known failure modes
Ten patterns were found by running the model on 32 constructed samples outside the dataset, each written to exercise one policy rule. Examples in full are in the project report. In summary: generic nouns labeled as named entities; vague time expressions labeled; boundaries falling inside words; single entities split into two differently labeled spans; entities missed; possessives dropped from establishment names; dish preparations labeled; nationality adjectives kept inside spans inconsistently; adjacent time expressions merged; adverbial phrases labeled as jobs.
At least one of these is inherited rather than learned wrongly. The training data itself contains nationality adjectives treated inconsistently, so the model reproduces that faithfully.
Related repositories
| Stage | Artefact |
|---|---|
| 1 | inovruzova/polygraf-ner-stage1-cleaned |
| 2 | inovruzova/polygraf-ner-stage2-expanded |
Licensing
The model weights are released under Apache 2.0, matching the base model.
The training data is a collection of corpora under different terms, including share-alike obligations from ODbL and CC BY-SA. Those obligations attach to the dataset rather than to a set of trained weights, but anyone reusing this model should read the dataset card.
Produced during the Polygraf Applied NLP / NER technical task.