Base-Ro-125M: Romanian Foundation Causal Language Model
Base-Ro-125M is a 124.8M-parameter causal language model based on the modern LLaMA architecture, trained from scratch ("Token Zero") purely on Romanian web text.
Model Architecture
| Parameter | Value |
|---|---|
| Architecture | LLaMA Causal Decoder (LlamaForCausalLM) |
| Parameters | 124,789,248 (~124.8M) |
| Hidden Size ($d_{\text{model}}$) | 768 |
| Intermediate Size (SwiGLU) | 2,048 |
| Layers | 12 |
| Attention Heads | 12 (Query) / 4 (Key-Value) — Grouped-Query Attention (GQA 3:1) |
| Positional Encoding | Rotary Position Embedding (RoPE) |
| Normalization | RMSNorm ($\epsilon = 10^{-5}$) |
| Context Length | 1,024 tokens |
| Vocabulary | 16,384 Byte-Pair Encoding (BPE) optimized for Romanian diacritics (ș, ț, ă, î, â) |
Pretraining Details
- Corpus: Cleaned Romanian web text comprising Romanian Wikipedia, 5-year news archives (HotNews, Digi24, G4Media), and FineWeb-2 Romanian web shards.
- Tokens Trained: ~3.27 Billion tokens (50,000 additional steps, total 60,000 steps with effective batch size 64 $\times$ 1,024 context).
- Hardware: Single NVIDIA GeForce RTX 3060 12GB GDDR6 (Mixed Precision BF16 / FlashAttention SDPA).
- Training Time: ~50 hours continuous pretraining.
- Final Pretraining Loss:
2.833| Perplexity:16.99.
Usage with Transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "flaviussteff/base-ro-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
device_map="auto",
)
prompt = "În România contemporană, dezvoltarea economică"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=60,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.15,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Evaluation Benchmark
Evaluated on the parallel Romanian-English socio-cultural bias benchmark across 37 counterfactual diagnostic pairs:
- Overall Stereotype Preference Metric ($SPM$):
56.76%(where 50.0% is neutral). - Gender & Occupation ($SPM$):
90.0%(absorbs prevailing web stereotypes regarding executive leadership, domestic roles, and caretaking). - Minority Framing ($SPM$):
62.5%.
Citation & Academic Context
Developed as part of academic research at the Faculty of Mathematics and Computer Science, University of Bucharest.