RuBERT-ruLaw
This model is a continued pretraining of DeepPavlov/rubert-base-cased
on the RusLawOD dataset — a large corpus of Russian legal texts (court decisions and normative acts).
The goal of this training is to improve RuBERT’s performance on legal domain tasks such as classification, information extraction, and retrieval.
Repository: https://github.com/TryDotAtwo/ruBERT-ruLaw
Training Details
- Base model: DeepPavlov/rubert-base-cased
- Task: Masked Language Modeling (MLM)
- Max sequence length: 512 tokens (stride 128)
- Batch size: 160 per device
- Gradient accumulation: 1
- Epochs: 8 (3 in test mode)
- Max steps: 40,000
- Warmup steps: 2,000
- Mixed precision: BF16 (on A100/H100)
- Optimizer & scheduler: Default Hugging Face Trainer settings
- Evaluation metric:
eval_loss(best checkpoint loaded at end) - Hardware: 3× NVIDIA H200 GPUs
- Final eval loss: < 0.3
Dataset
We use the RusLawOD dataset.
Before tokenization, empty or None entries are filtered out. Tokenization is performed with the RuBERT tokenizer with truncation and sliding window (stride 128) to maximize coverage of long documents.
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_name = "TryDotAtwo/rubert-rulaw"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
Evaluation Overview
Models were tested on the sud-resh-benchmark legal texts using a masked language modeling setup. Tokens were randomly masked at varying probabilities (10–40%), and models predicted them using their pre-trained heads.
Note: The ruBERT-ruLaw model was pre-trained on legal texts such as laws and statutes, but not specifically on judicial decisions. The evaluation reflects how well it generalizes to predicting masked tokens in Russian court rulings.
- Top-1 Accuracy: fraction of masked tokens predicted exactly.
- Top-5 Accuracy: fraction of masked tokens predicted within the top 5 candidates.
Results reflect performance across all masked tokens, aggregated for the dataset.
MLM Accuracy Comparison
| MLM Probability | Metric | ruBERT-ruLaw | rubert-base-cased | legal-bert-base-uncased |
|---|---|---|---|---|
| 10% | Top-1 | 81.0% | 73.0% | 45.3% |
| 10% | Top-5 | 92.2% | 87.0% | 77.2% |
| 15% | Top-1 | 78.8% | 67.9% | 45.3% |
| 15% | Top-5 | 90.8% | 83.2% | 76.7% |
| 20% | Top-1 | 76.3% | 53.8% | 45.0% |
| 20% | Top-5 | 89.0% | 71.5% | 75.9% |
| 25% | Top-1 | 73.6% | 18.0% | 44.4% |
| 25% | Top-5 | 87.0% | 31.9% | 75.0% |
| 30% | Top-1 | 70.4% | 5.9% | 43.8% |
| 30% | Top-5 | 84.6% | 10.9% | 74.0% |
| 35% | Top-1 | 66.9% | 6.0% | 42.9% |
| 35% | Top-5 | 81.9% | 9.1% | 72.9% |
| 40% | Top-1 | 62.9% | 6.0% | 41.9% |
| 40% | Top-5 | 78.5% | 8.5% | 71.7% |
Citation
A paper describing the dataset and training process will be released on arXiv soon. [Link — coming soon]