Estonian summariser
Rahvusarhiiv/et_summariser is an experimental Estonian summarization model fine-tuned from facebook/mbart-large-50-many-to-many-mmt.
This is a research release. It is not claimed to be state of the art; the model should be evaluated for the target domain before production use.
Intended Use
- Summarization of Estonian news, legal/administrative, encyclopedic, or academic-style text.
- Batch/offline summarization, evaluation, and retrieval-assisted workflows where outputs can be checked.
- Not intended as a sole source of factual truth, legal advice, medical advice, or archival description without human review.
How to Use
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
repo_id = "Rahvusarhiiv/et_summariser"
text = "Riigikogu arutas täna riigieelarve muudatusi. Valitsuse esindajad rõhutasid vajadust hoida kulud kontrolli all, opositsioon aga kritiseeris kärbete mõju haridusele ja kohalikele omavalitsustele."
tokenizer = AutoTokenizer.from_pretrained(repo_id, src_lang="et_EE")
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=768)
language_token_id = tokenizer.convert_tokens_to_ids("et_EE")
summary_ids = model.generate(
**inputs,
forced_bos_token_id=language_token_id,
max_new_tokens=512, num_beams=4, no_repeat_ngram_size=3,
)
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0])
Recommended Generation Settings
max_new_tokens:512num_beams:4no_repeat_ngram_size:3
Training Data
Fine-tuned from mBART-50 on cleaned Estonian summarization data, including ERR-style news, LongSum-style long-form examples, and Estonian EurLex/legal material. Grounding filters, metadata dropout, and evidence selection were used during preprocessing.
- Estonian news and long-form summarization corpora prepared locally.
- Estonian EurLex/legal examples after boilerplate filtering.
- Held-out Wiki4 and GDELT-style news evaluations were not used for training.
Known public Hub datasets used or partially used where applicable:
- No public Hub dataset metadata is declared for local-only collected datasets.
Training metadata:
- Base/initial model recorded by trainer:
facebook/mbart-large-50-many-to-many-mmt - Train rows:
23996 - Validation rows:
1000 - Max source tokens:
768 - Max target tokens:
512 - Seed:
20260611
Evaluation
| benchmark | n | ROUGE-L | ROUGE-1 | ROUGE-2 | ref coverage | src precision | judge factuality | judge coverage | judge overall | words | repeat 4g | legalese | max hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Wiki4 strict v2 LLM judge | 100 | 3.01 | 3.33 | 2.73 | |||||||||
| Wiki4 broad LLM judge | 200 | 3.02 | 2.42 | 2.33 | |||||||||
| Wiki4 final automatic | 100 | 0.294 | 0.351 | 0.201 | 0.387 | 0.751 | 62.4 | 0.0000 | 0.0000 | 0.0000 | |||
| GDELT final automatic | 100 | 0.169 | 0.212 | 0.075 | 0.257 | 0.805 | 47.5 | 0.0000 | 0.0000 | 0.0000 |
The automatic metrics include ROUGE, coverage proxies, repetition and artifact checks. The judge metrics use a local Ollama LLM judge on sampled examples and should be treated as comparative, not absolute.
Limitations and Failure Modes
- The model can hallucinate or over-generalize details, especially when the input is long or under-specified.
- Named entities, numbers, dates, and legal references should be verified against the source.
- Some evaluation datasets reward lead-like summaries; performance may differ on non-news or conversational text.
- The training data mixes sources with different writing styles and upstream terms; users are responsible for checking suitability for their use case.
- Qualitative review still found occasional unsupported details, so factual verification is recommended for publication workflows.
Bias, Risks, and Ethical Considerations
The model reflects the distributions and editorial choices of its source corpora. It may reproduce biases present in news, legal, academic, or synthetic summaries. Human review is recommended for public-facing summaries and high-impact decisions.
CO2 Emissions
Training emissions were not instrumented with CodeCarbon or an equivalent tracker. No numeric CO2 estimate is reported. Training was local fine-tuning of an mBART-50 checkpoint rather than pretraining from scratch.
Model Tree
- Base model:
facebook/mbart-large-50-many-to-many-mmt - Relationship: fine-tuned sequence-to-sequence summarization checkpoint.
License
The model card uses license: other because the training mix contains multiple upstream data sources with different terms. Treat this as a research-use release and verify upstream data/license constraints before redistribution or commercial use.