TahrirchiBERT small model

The TahrirchiBERT-small is an encoder-only Transformer text model with 67 million parameters. It is pretrained model on Uzbek language (latin script) using a masked language modeling (MLM) objective. This model is case-sensitive: it does make a difference between uzbek and Uzbek.

For full details of this model please read our paper (coming soon!) and release blog post.

Model variations

This model is part of the family of TahrirchiBERT models trained with different number of parameters that will continuously expanded in the future.

Intended uses & limitations

This model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering.

How to use

You can use this model directly with a pipeline for masked language modeling:

>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='tahrirchi/tahrirchi-bert-small')
>>> unmasker("Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning <mask>, mutafakkiri va davlat arbobi bo‘lgan.")

[{'score': 0.24358688294887543,
  'token': 10374,
  'token_str': ' shoiri',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning shoiri, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.1781526654958725,
  'token': 11766,
  'token_str': ' asoschisi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning asoschisi, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.06118808686733246,
  'token': 14718,
  'token_str': ' olimi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning olimi, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.05062260851264,
  'token': 3360,
  'token_str': ' buyuk',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning buyuk, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.03740185126662254,
  'token': 16300,
  'token_str': ' atoqli',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning atoqli, mutafakkiri va davlat arbobi bo‘lgan.'}]


>>> unmasker("Egiluvchan boʻgʻinlari va <mask>, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.")

[{'score': 0.1697940230369568,
  'token': 1376,
  'token_str': ' shuningdek',
  'sequence': 'Egiluvchan boʻgʻinlari va shuningdek, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.030622979626059532,
  'token': 3359,
  'token_str': ' ayniqsa',
  'sequence': 'Egiluvchan boʻgʻinlari va ayniqsa, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.02887713722884655,
  'token': 10013,
  'token_str': ' daraxtlar',
  'sequence': 'Egiluvchan boʻgʻinlari va daraxtlar, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.02008388563990593,
  'token': 5038,
  'token_str': ' xususan',
  'sequence': 'Egiluvchan boʻgʻinlari va xususan, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.018601732328534126,
  'token': 30498,
  'token_str': ' butalar',
  'sequence': 'Egiluvchan boʻgʻinlari va butalar, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'}]

Training data

TahrirchiBERT is pretrained using a standard Masked Language Modeling (MLM) objective: the model is given a sequence of text with some tokens hidden, and it has to predict these masked tokens. TahrirchiBERT is trained on the Uzbek Crawl and small high quality latin portion of Uzbek Books, which contains roughly 4000 preprocessd books, 1.2 million curated text documents scraped from the internet and Telegram blogs.

Training procedure

Preprocessing

The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 30,528 to make fully use of rare words. The inputs of the model take pieces of 512 contiguous tokens that may span over documents. Also, we added number of regular expressions to avoid misrepresentation of different symbols that is used mostly incorrectly in practise.

Pretraining

The model was trained for one million steps with a batch size of 512. The sequence length was limited to 512 tokens during all pre-training stage. The optimizer used is Adam with a learning rate of 5e-4, \(\beta_{1} = 0.9\) and \(\beta_{2} = 0.98\), a weight decay of 1e-5, learning rate warmup to the full LR for 6% of the training duration with linearly decay to 0.02x the full LR by the end of the training duration.

Citation

Please cite this model using the following format:

@online{Mamasaidov2023TahrirchiBERT,
    author    = {Mukhammadsaid Mamasaidov and Abror Shopulatov},
    title     = {TahrirchiBERT small},
    year      = {2023},
    url       = {https://huggingface.co/tahrirchi/tahrirchi-bert-small},
    note      = {Accessed: 2023-10-27}, % change this date
    urldate   = {2023-10-27} % change this date
}

Gratitude

We are thankfull for these awesome organizations and people for help to make it happen:

• MosaicML team: for their script for efficiently training BERT models
• Ilya Gusev: for advise throughout the process
• David Dale: for advise throughout the process