IndicBERT
IndicBERT is a multilingual ALBERT model pretrained exclusively on 12 major Indian languages. It is pre-trained on our novel monolingual corpus of around 9 billion tokens and subsequently evaluated on a set of diverse tasks. IndicBERT has much fewer parameters than other multilingual models (mBERT, XLM-R etc.) while it also achieves a performance on-par or better than these models.
The 12 languages covered by IndicBERT are: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.
The code can be found here. For more information, checkout our project page or our paper.
Pretraining Corpus
We pre-trained indic-bert on AI4Bharat's monolingual corpus. The corpus has the following distribution of languages:
| Language | as | bn | en | gu | hi | kn | |
|---|---|---|---|---|---|---|---|
| No. of Tokens | 36.9M | 815M | 1.34B | 724M | 1.84B | 712M | |
| Language | ml | mr | or | pa | ta | te | all |
| No. of Tokens | 767M | 560M | 104M | 814M | 549M | 671M | 8.9B |
Evaluation Results
IndicBERT is evaluated on IndicGLUE and some additional tasks. The results are summarized below. For more details about the tasks, refer our official repo
IndicGLUE
| Task | mBERT | XLM-R | IndicBERT |
|---|---|---|---|
| News Article Headline Prediction | 89.58 | 95.52 | 95.87 |
| Wikipedia Section Title Prediction | 73.66 | 66.33 | 73.31 |
| Cloze-style multiple-choice QA | 39.16 | 27.98 | 41.87 |
| Article Genre Classification | 90.63 | 97.03 | 97.34 |
| Named Entity Recognition (F1-score) | 73.24 | 65.93 | 64.47 |
| Cross-Lingual Sentence Retrieval Task | 21.46 | 13.74 | 27.12 |
| Average | 64.62 | 61.09 | 66.66 |
Additional Tasks
| Task | Task Type | mBERT | XLM-R | IndicBERT |
|---|---|---|---|---|
| BBC News Classification | Genre Classification | 60.55 | 75.52 | 74.60 |
| IIT Product Reviews | Sentiment Analysis | 74.57 | 78.97 | 71.32 |
| IITP Movie Reviews | Sentiment Analaysis | 56.77 | 61.61 | 59.03 |
| Soham News Article | Genre Classification | 80.23 | 87.6 | 78.45 |
| Midas Discourse | Discourse Analysis | 71.20 | 79.94 | 78.44 |
| iNLTK Headlines Classification | Genre Classification | 87.95 | 93.38 | 94.52 |
| ACTSA Sentiment Analysis | Sentiment Analysis | 48.53 | 59.33 | 61.18 |
| Winograd NLI | Natural Language Inference | 56.34 | 55.87 | 56.34 |
| Choice of Plausible Alternative (COPA) | Natural Language Inference | 54.92 | 51.13 | 58.33 |
| Amrita Exact Paraphrase | Paraphrase Detection | 93.81 | 93.02 | 93.75 |
| Amrita Rough Paraphrase | Paraphrase Detection | 83.38 | 82.20 | 84.33 |
| Average | 69.84 | 74.42 | 73.66 |
* Note: all models have been restricted to a max_seq_length of 128.
Downloads
The model can be downloaded here. Both tf checkpoints and pytorch binaries are included in the archive. Alternatively, you can also download it from Huggingface.
Citing
If you are using any of the resources, please cite the following article:
@inproceedings{kakwani2020indicnlpsuite,
title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
author={Divyanshu Kakwani and Anoop Kunchukuttan and Satish Golla and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
year={2020},
booktitle={Findings of EMNLP},
}
We would like to hear from you if:
- You are using our resources. Please let us know how you are putting these resources to use.
- You have any feedback on these resources.
License
The IndicBERT code (and models) are released under the MIT License.
Contributors
- Divyanshu Kakwani
- Anoop Kunchukuttan
- Gokul NC
- Satish Golla
- Avik Bhattacharyya
- Mitesh Khapra
- Pratyush Kumar
This work is the outcome of a volunteer effort as part of AI4Bharat initiative.
Contact
- Anoop Kunchukuttan (anoop.kunchukuttan@gmail.com)
- Mitesh Khapra (miteshk@cse.iitm.ac.in)
- Pratyush Kumar (pratyush@cse.iitm.ac.in)