Knesset-DictaBERT
Knesset-DictaBERT is a Hebrew language model fine-tuned on the Knesset Corpus, which comprises Israeli parliamentary proceedings.
This model is based on the Dicta-BERT architecture and is designed to understand and generate text in Hebrew, with a specific focus on parliamentary language and context.
Model Details
- Model type: BERT-based (Bidirectional Encoder Representations from Transformers)
- Language: Hebrew
- Training Data: Knesset Corpus (Israeli parliamentary proceedings)
- Base Model: Dicta-BERT
Training Procedure
The model was fine-tuned using the masked language modeling (MLM) task on the Knesset Corpus. The MLM task involves predicting masked words in a sentence, allowing the model to learn contextual representations of words.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("GiliGold/Knesset-DictaBERT")
model = AutoModelForMaskedLM.from_pretrained("GiliGold/Knesset-DictaBERT")
model.eval()
sentence = "יש לנו [MASK] על זה בשבוע הבא"
# Tokenize the input sentence and get predictions
inputs = tokenizer.encode(sentence, return_tensors='pt')
output = model(inputs)
mask_token_index = 3
top_2_tokens = torch.topk(output.logits[0, mask_token_index, :], 2)[1]
# Convert token IDs to tokens and print them
print('\n'.join(tokenizer.convert_ids_to_tokens(top_2_tokens)))
# Example output: ישיבה / דיון
Evaluation
The evaluation was conducted on a 10% test set of the Knesset Corpus, consisting of approximately 3.2 million sentences. The perplexity was calculated on this full test set. Due to time constraints, accuracy measures were calculated on a subset of this test set, consisting of approximately 300,000 sentences (approximately 3.5 million tokens).
Perplexity
The perplexity of the original DictaBERT on the full test set is 22.87.
The perplexity of Knesset-DictaBERT on the full test set is 6.60.
Accuracy
- 1-accuracy results
Knesset-DictaBERT identified the correct token in the top-1 prediction in 52.55% of the cases.
The original DictaBERT model achieved a top-1 accuracy of 48.02%.
- 2-accuracy results
Knesset-DictaBERT identified the correct token within the top-2 predictions in 63.07% of the cases.
The original DictaBERT model achieved a top-2 accuracy of 58.60%.
- 5-accuracy results
Knesset-DictaBERT identified the correct token within the top-5 predictions in 73.59% of the cases.
The original DictaBERT model achieved a top-5 accuracy of 68.98%.
Acknowledgments
This model is built upon the work of the Dicta team, and their contributions are gratefully acknowledged.
Citation
If you use this model in your work, please cite:
@misc{goldin2024knessetdictaberthebrewlanguagemodel,
title={Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings},
author={Gili Goldin and Shuly Wintner},
year={2024},
eprint={2407.20581},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2407.20581},
}