ViLegalBERT-ext
ViLegalBERT-ext is a Vietnamese legal-domain encoder language model obtained via continual pretraining of Qualcomm-AI-Research/BamiBERT on a large, cleaned, deduplicated corpus of Vietnamese legal text.
The name "ext" refers to the two key limits this model extends relative to a previous model - ViLegalBERT:
- Context length: 256 tokens → 2048 tokens
- Legal pretraining corpus size: 16GB → 20.57GB (~7.15B legal-domain tokens)
Unlike PhoBERT-based models, ViLegalBERT-ext requires no external Vietnamese word segmentation as a preprocessing step — it operates directly on raw text via byte-level BPE tokenizer.
Model Details
- Architecture: BERT-base (12 Transformer layers), RoBERTa-style masked language modeling objective (dynamic masking, no next-sentence-prediction)
- Base model:
Qualcomm-AI-Research/BamiBERT(~103M parameters), itself trained from scratch on 129GB of general-domain Vietnamese text - Tokenizer: Byte-level BPE extended from PhoGPT's Vietnamese tokenizer, vocab size 20,481 (inherited unchanged from BamiBERT)
- Max sequence length: 2048 tokens
- Precision: trained in BF16 mixed precision
Continual Pretraining Data
- Corpus: Vietnamese legal text
- Size: 20.57GB of raw legal text, tokenized and packed (~7.15B legal-domain tokens)
Training Procedure
- Objective: Masked language modeling with dynamic masking (15% mask probability), consistent with BamiBERT's original RoBERTa-style pretraining recipe
- Optimizer: AdamW
- Learning rate: 2e-5 (peak), linear schedule with warmup — chosen conservatively (~1/7 of BamiBERT's original 1.5e-4 peak LR) to reduce the risk of catastrophic forgetting during domain-adaptive continual pretraining
- Effective batch size: 144 (per-device batch size 48 × gradient accumulation 3)
- Hardware: single NVIDIA H100 80GB GPU
Intended Use
ViLegalBERT-ext is an encoder model intended for fill-mask and as a base for fine-tuning on downstream Vietnamese legal NLP tasks, such as:
- Legal document classification
- Legal named entity recognition
- Legal question answering / retrieval (as an embedding backbone)
- Legal text similarity / clause matching
It is not designed for open-ended text generation.
Evaluation
Evaluation results (downstream task benchmarks) coming soon.
Limitations
- Inherits any limitations of the base BamiBERT model, including potential underperformance on non-standard Vietnamese dialects and general temporal concept drift.
- Continual pretraining was conducted on legal text; general-domain qualitative checks did not reveal meaningful catastrophic forgetting, but no comprehensive general-domain benchmark evaluation has been run.
License
This model is a derivative of Qualcomm-AI-Research/BamiBERT and is released under the same license terms: BSD 3-Clause Clear and the Qualcomm Responsible AI License. See the base model's license for full terms.
Citation
If you use this model, please cite the original BamiBERT model.
@article{BamiBERT,
title = {{BamiBERT: A New BERT-based Language Model for Vietnamese}},
author = {Dat Quoc Nguyen and Thinh Pham and Chi Tran and Linh The Nguyen},
journal = {arXiv preprint},
volume = {arXiv:2607.02259},
year = {2026}
}