Persian ZWNJ-aware ALBERT MLM
ALBERT-base-v2 adapted to Persian with a freshly trained 38,016-token
SentencePiece unigram tokenizer. English Wikipedia supplies additional English
text. ZWNJ (U+200C, Persian half-space) is preserved as a standalone token, ID 5.
Maximum input length: 512 tokens.
Training
Initialized from albert/albert-base-v2,
with word embeddings and the tied vocabulary head reinitialized for the new
vocabulary. Trained with 15% masked-language modeling on normalized
shekar-ai/Shiraz and English
Wikipedia (target 10% of corpus characters). Persian text is normalized with
shekar.Normalizer before tokenization; the tokenizer itself only normalizes
whitespace. The tokenizer was trained from up to 6 million sampled corpus lines.
Released checkpoint: 134,000 optimizer steps, 2.20 epochs, stopped early as validation gains diminished. Effective batch size 128, BF16, two RTX A6000 GPUs. The training dataset contains 7,794,452 examples: 75% packed 512-token blocks and 25% natural 8–64-token sentences. Natural short sentences stay intact; 10% of long blocks are randomly cropped. Sentence-content hashes separate train/eval before packing; this is sentence-level, not document-level separation.
Held-out MLM loss at the released checkpoint (lower is better):
| Full blocks | Natural short sentences | 64-token crops |
|---|---|---|
| 2.359 | 2.422 | 2.591 |
Usage
from transformers import pipeline
from shekar import Normalizer
fill = pipeline(
"fill-mask",
model="shekar-ai/albert-base-v2-persian-zwnj-shiraz-mlm",
device=-1, # CPU
)
text = Normalizer()("سلام بر تو ای") + " [MASK]"
print(fill(text, top_k=5))
Install torch, transformers, and shekar for this example. Use [MASK]
(uppercase). Each mask predicts one tokenizer token, which may be a word piece.
Compatibility and limitations
Tested with Transformers 4.57.6 / tokenizers 0.22.2 and
Transformers 5.17.0 / tokenizers 0.23.2: identical token IDs on the test texts,
tokenizer save/reload, preserved ZWNJ, tied embeddings, and finite CPU outputs
at 8, 64, 511, and 512 tokens. See compatibility.json.
No custom model code or trust_remote_code is required.
This is an MLM checkpoint for fill-mask and further fine-tuning. It has not yet been contrastively trained or benchmarked as a sentence embedding model. Predictions can be inaccurate or reflect corpus biases; English capability is limited and has not been established by a comprehensive benchmark.
License
This adapted checkpoint and fresh tokenizer are released under the MIT License
(see LICENSE). The original ALBERT base is Apache-2.0 licensed; its license is
included in LICENSE.base-albert. Original upstream rights and notices remain
applicable to the base components.