Estonian NER model based on EstBERT

This model is a fine-tuned version of tartuNLP/EstBERT on the Estonian NER dataset. The model was trained by tartuNLP, the NLP research group at the institute of Computer Science at the University of Tartu.

It achieves the following results on the test set:

• Loss: 0.3565
• Precision: 0.7612
• Recall: 0.7744
• F1: 0.7678
• Accuracy: 0.9672

The entity-level results are as follows:

How to use

You can use this model with Transformers pipeline for NER. Post-processing of results may be necessary as the model occasionally tags subword tokens as entities.

from transformers import BertTokenizer, BertForTokenClassification
from transformers import pipeline

tokenizer = BertTokenizer.from_pretrained('tartuNLP/EstBERT_NER')
bertner = BertForTokenClassification.from_pretrained('tartuNLP/EstBERT_NER')

nlp = pipeline("ner", model=bertner, tokenizer=tokenizer)

text = "Kaia Kanepi (WTA 57.) langes USA-s Charlestonis toimuval WTA 500 kategooria tenniseturniiril konkurentsist kaheksandikfinaalis, kaotades poolatarile Magda Linette'ile (WTA 64.) 3 : 6, 6 : 4, 2 : 6."

ner_results = nlp(text)

tokens=tokenizer(text)
tokens=tokenizer.convert_ids_to_tokens(tokens['input_ids'])


print(f'tokens: {tokens}')
print(f'NER model:{ner_results}')

tokens: ['[CLS]', 'kai', '##a', 'kanepi', '(', 'w', '##ta', '57', '.', ')', 'langes', 'usa', '-', 's', 'cha', '##rl', '##est', '##onis', 'toimuval', 'w', '##ta', '500', 'kategooria', 'tennise', '##turniiril', 'konkurentsist', 'kaheksandik', '##finaalis', ',', 'kaotades', 'poola', '##tari', '##le', 'ma', '##gda', 'line', '##tte', "'", 'ile', '(', 'w', '##ta', '64', '.', ')', '3', ':', '6', ',', '6', ':', '4', ',', '2', ':', '6', '.', '[SEP]']

NER model: [{'entity': 'B-PER', 'score': 0.99999887, 'index': 1, 'word': 'kai', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.97371966, 'index': 2, 'word': '##a', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999815, 'index': 3, 'word': 'kanepi', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.63085276, 'index': 5, 'word': 'w', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.99999934, 'index': 11, 'word': 'usa', 'start': None, 'end': None}, {'entity': 'B-GPE', 'score': 0.9999685, 'index': 14, 'word': 'cha', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.8875574, 'index': 15, 'word': '##rl', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9996168, 'index': 16, 'word': '##est', 'start': None, 'end': None}, {'entity': 'I-GPE', 'score': 0.9992657, 'index': 17, 'word': '##onis', 'start': None, 'end': None}, {'entity': 'B-EVENT', 'score': 0.99999064, 'index': 19, 'word': 'w', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.9772493, 'index': 20, 'word': '##ta', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99999076, 'index': 21, 'word': '500', 'start': None, 'end': None}, {'entity': 'I-EVENT', 'score': 0.99955636, 'index': 22, 'word': 'kategooria', 'start': None, 'end': None}, {'entity': 'B-TITLE', 'score': 0.8771319, 'index': 30, 'word': 'poola', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.99999785, 'index': 33, 'word': 'ma', 'start': None, 'end': None}, {'entity': 'B-PER', 'score': 0.9998398, 'index': 34, 'word': '##gda', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999987, 'index': 35, 'word': 'line', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999976, 'index': 36, 'word': '##tte', 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.99999285, 'index': 37, 'word': "'", 'start': None, 'end': None}, {'entity': 'I-PER', 'score': 0.9999794, 'index': 38, 'word': 'ile', 'start': None, 'end': None}, {'entity': 'B-ORG', 'score': 0.7664479, 'index': 40, 'word': 'w', 'start': None, 'end': None}]

Intended uses & limitations

This model can be used to find named entities from Estonian texts. The model is free to use for anyone. TartuNLP does not guarantee that the model is useful for anyone or anything. TartuNLP is not responsible for any results it generates.

Training and evaluation data

The model was trained on two Estonian NER datasets:

• The Reannotated Estonian NER corpus
• The New Estonian NER corpus

Both datasets have been annotated with the same annotation scheme. For training this model, the datasets were joined.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 1024
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
• lr_scheduler_type: polynomial
• max num_epochs: 150
• early stopping limit: 20
• early stopping tol: 0.0001
• mixed_precision_training: Native AMP

Training results

The final model was saved after epoch 53 (shown in bold) where the overall F1 was the highest on the development set.

Framework versions

• Transformers 4.16.2
• Pytorch 1.10.2+cu113
• Datasets 1.18.3
• Tokenizers 0.11.0

BibTeX entry and citation info

@inproceedings{tanvir-etal-2021-estbert,
    title = {{EstBERT: A Pretrained Language-Specific BERT for Estonian}},
    author = "Tanvir, Hasan  and
      Kittask, Claudia  and
      Eiche, Sandra  and
      Sirts, Kairit",
    editor = "Dobnik, Simon  and
      {\O}vrelid, Lilja",
    booktitle = "Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)",
    year = "2021",
    address = "Reykjavik, Iceland (Online)",
    publisher = {Link{\"o}ping University Electronic Press, Sweden},
    url = "https://aclanthology.org/2021.nodalida-main.2/",
    pages = "11--19"
}

@misc{tanvir2020estbert,
      title={EstBERT: A Pretrained Language-Specific BERT for Estonian}, 
      author={Hasan Tanvir and Claudia Kittask and Kairit Sirts},
      year={2020},
      eprint={2011.04784},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}