language: id license: mit base_model: indobenchmark/indobert-base-p1 pipeline_tag: fill-mask tags:
- bert
- indonesian
- islamic
- quran
- hadith
- masked-language-modeling
- fill-mask library_name: transformers
ALBERTIR: BERT-Based Pretraining for Indonesian Religious Texts
Model Description
ALBERTIR is a domain-specific Indonesian language model pretrained on Islamic texts (Quran and Hadith translations). It is built upon the indobenchmark/indobert-base-p1 architecture and further pretrained using Masked Language Modeling (MLM) on an Islamic corpus.
This model is designed for:
- Semantic alignment of Indonesian Islamic texts
- Theological accuracy evaluation
- Indonesian Friday sermon (Khutbah) analysis
- Islamic content understanding and classification
Developed by
- Irwan Darmawan – Universitas Negeri Malang, Indonesia
- Hakkun Elmunsyah – Universitas Negeri Malang, Indonesia
- Didik Dwi Prasetya – Universitas Negeri Malang, Indonesia
- Wahyu Mulyo Utomo – Universiti Tun Hussein Onn Malaysia, Malaysia
Model Details
| Parameter | Value |
|---|---|
| Model Type | BERT (Bidirectional Encoder Representations from Transformers) |
| Base Model | indobenchmark/indobert-base-p1 |
| Task | Masked Language Modeling (MLM) |
| Language | Indonesian |
| Domain | Islamic / Religious Texts |
| Parameters | 124,492,880 |
| Tensor Type | F32 |
Pretraining Details
| Parameter | Value |
|---|---|
| Corpus | Quran (6,236 verses) + Sahih Bukhari (7,008) + Sahih Muslim (5,362) |
| Total Documents | 18,606 |
| Epochs | 10 |
| Batch Size | 8 |
| Learning Rate | 5e-5 |
| Optimizer | AdamW |
| Total Steps | 10,470 |
| Final Training Loss | ~1.2473 |
| Final Validation Loss | ~1.1415 |
Training Data
The model was trained on:
- Quran: Indonesian translation (Ministry of Religious Affairs)
- Sahih Bukhari: Indonesian translation
- Sahih Muslim: Indonesian translation
Total: 18,606 verses/hadith documents
Intended Uses
- Fill-mask tasks on Indonesian Islamic texts
- Semantic similarity in religious domain
- Theological content classification
- Friday sermon analysis
How to Use
from transformers import AutoModelForMaskedLM, AutoTokenizer
model_id = "irwan19/albertir-quran-hadith"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
# Example: Fill-mask prediction
text = "Umat Islam wajib melaksanakan ibadah [MASK] di bulan Ramadhan."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# Prediksi: 'puasa'