name
A bert-base-uncased encoder with a token-classification head, fine-tuned on CoNLL-2003 for named entity recognition. It assigns a label to each token and reports the spans it considers to be people, organisations, locations and miscellaneous entities. Training ran for three epochs at a learning rate of 2e-5 with batch size eight and a linear schedule with 500 warmup steps, a validation loss of 0.0474 as recorded during that run. Recomputing the loss over the full 3,250-sentence CoNLL-2003 validation split here gives 0.0524, close enough to confirm the figure while differing by about 0.005, which is the scale expected from differences in tokenisation and label alignment between runs.
An earlier version of this repository carried a generic label map, LABEL_0 through LABEL_8, so pipeline output reported LABEL_3 instead of an entity type. The label map in config.json now holds the CoNLL names in their canonical order: O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC. That ordering was checked against the model rather than assumed: on four held-out sentences, people, organisations and locations all resolve to the expected entity type, and European Union comes back B-ORG/I-ORG. The repository also no longer carries a duplicate pytorch_model.bin, a training_args.bin, or a runs/ directory of TensorBoard events, none of which affected inference.
Usage
from transformers import pipeline
ner = pipeline("token-classification", model="harpertoken/name", aggregation_strategy="simple")
for entity in ner("Angela Merkel visited the European Union"):
print(entity["word"], entity["entity_group"])
Evaluation
On the CoNLL-2003 test split of 3,453 sentences, 46,427 tokens and 5,648 entities, the model reaches entity-level precision 0.896, recall 0.909 and F1 0.902. Per type, F1 runs 0.968 for PER, 0.924 for LOC, 0.867 for ORG and 0.788 for MISC; MISC is the weak class, as it is in most CoNLL-2003 results, since that bucket absorbs anything the annotators did not classify as a person, organisation or place. Those figures are in line with published bert-base-uncased results on this benchmark.
One caveat on the measurement. The label map fixed above was verified on four hand-picked sentences, which is a weak test, so the full test split was used to confirm it properly. The evaluation data stores its tags as bare integers, and those integers are not in the canonical order: tag 1 is B-ORG rather than B-PER. Scoring under the assumed ordering yields F1 0.225, and under the ordering recovered from the model's own predictions, 0.902. That recovered mapping was afterwards checked against the dataset's own label.json, and it agrees on all nine classes, so the 0.902 figure rests on the right labels. Anyone repeating this evaluation should read that file before comparing numbers.
Limitations
CoNLL-2003 consists of English newswire from 2003, and entity boundaries in that data reflect contemporary editorial conventions. The model should not be expected to transfer to other languages, to literary or conversational prose, or to domains with unfamiliar entity types. Case is discarded by the uncased vocabulary, so predictions are returned lowercased and multi-word entities arrive split into subwords: Merkel becomes mer and kel, both labelled I-PER. Use aggregation_strategy="simple" if you want them reassembled.
Attribution
BERT follows Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019). CoNLL-2003 is described in CoNLL-2003 shared task papers from Nanyeong Bang et al. and Tjong Kim Sang et al.