mBART50 XL-Sum

mbart50-xlsum is a multilingual abstractive news summarization model fine-tuned from facebook/mbart-large-50-many-to-many-mmt on 15 language subsets of GEM/xlsum. It was developed as part of a multilingual news aggregation and summarization project.

This checkpoint is the first of two mBART-50 training variants produced by the project. It uses a standard multilingual mixture, a 96-token target limit, and a compute-conscious two-epoch training setup. Compared with the second variant, it is slightly more source-grounded in the project's automatic diagnostics, while producing very similar reference-overlap scores.

Related Project

This model was developed as part of samurAI, a multilingual news aggregation and summarization system with RSS ingestion, local transformer summarization, SQLite storage, a Flask web interface, JSON APIs, and evaluation notebooks.

Related checkpoint: mskayacioglu/mbart-xlsum-2

Model Details

Languages

The fine-tuning mixture contains the following 15 languages:

Quantitative evaluation was completed for 11 of these languages: Arabic, Chinese (Simplified), English, French, Hindi, Japanese, Korean, Russian, Spanish, Turkish, and Vietnamese. Results for Indonesian, Persian, Portuguese, and Ukrainian are not reported in the final common evaluation, so performance claims should not be extrapolated to them.

Intended Use

The model is intended for:

• summarizing news articles in a supported source language;
• research on multilingual abstractive summarization;
• comparing multilingual training and language-balancing strategies;
• integration into news-processing pipelines with human review.

The model is not intended to be a factual authority, to make editorial decisions autonomously, or to summarize high-stakes medical, legal, financial, or emergency information without verification against the source.

Training Data

The checkpoint was trained on GEM/xlsum, a multilingual collection of BBC news articles paired with short reference summaries. Examples from the 15 selected language subsets were combined, shuffled, and annotated with their corresponding mBART language codes.

The final benchmark below is separate from these training-time subsets and uses the full executable test subsets from csebuetnlp/xlsum.

Training Procedure

Generation-based validation was disabled during training to control compute cost, and no best-checkpoint selection was performed. Training completed 6,390 optimization steps with a final reported training loss of 2.2078 over approximately 4 hours 44 minutes on CUDA hardware.

Evaluation

Protocol

The final evaluation used the complete available test subsets of csebuetnlp/xlsum for 11 languages, totaling 52,219 examples per model. No per-language sample cap was applied. Scores below are macro-averaged across languages and reported on a 0-1 scale.

Articles were truncated to 512 tokens. Evaluation used batch size 8 and mixed precision on CUDA. The common mBART decoding configuration was:

max_length=90
min_length=24
num_beams=5
length_penalty=1.1
no_repeat_ngram_size=3
repetition_penalty=1.05
early_stopping=True

Turkish used a language-specific configuration with max_length=84, min_length=26, 6 beams, length penalty 1.3, 4-gram repetition blocking, and repetition penalty 1.2. The target language token was forced for every language.

Overall Results

METEOR-lite is the project's lightweight exact-token implementation and must not be interpreted as the canonical METEOR package score.

Results by Language

Supplementary Diagnostics

These diagnostics are included to describe model behavior, not as established factuality or human-quality measures.

Latency was measured in one CUDA evaluation environment and is not portable across hardware, software versions, or batch configurations. The completeness and factuality values are project-specific automatic proxies; they do not prove semantic completeness or factual correctness.

Usage

import torch
from transformers import MBart50TokenizerFast, MBartForConditionalGeneration

model_id = "mskayacioglu/mbart50-xlsum"
language_code = "tr_TR"

tokenizer = MBart50TokenizerFast.from_pretrained(model_id)
model = MBartForConditionalGeneration.from_pretrained(model_id)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

article = "Özetlenecek Türkçe haber metni buraya yazılır."
tokenizer.src_lang = language_code

inputs = tokenizer(
    article,
    return_tensors="pt",
    max_length=512,
    truncation=True,
).to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.lang_code_to_id[language_code],
        max_length=84,
        min_length=26,
        num_beams=6,
        length_penalty=1.3,
        no_repeat_ngram_size=4,
        repetition_penalty=1.2,
        early_stopping=True,
    )

summary = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
print(summary)

For languages other than Turkish, the evaluation configuration used max_length=90, min_length=24, 5 beams, length penalty 1.1, 3-gram repetition blocking, and repetition penalty 1.05.

Limitations and Risks

• The model can hallucinate facts, names, dates, quantities, or causal relationships.
• Inputs longer than 512 tokens were truncated during fine-tuning and evaluation; important information near the end of long articles may be omitted.
• XL-Sum is based on BBC news. Performance may decline on other publishers, informal writing, specialist domains, or structurally noisy web text.
• Automatic overlap metrics reward lexical similarity and do not establish factual consistency or summary usefulness.
• Evaluation coverage is uneven across languages and includes only 11 languages in the final common benchmark.
• The training mixture is imbalanced across languages, so high-resource languages can influence the model more strongly.
• The model may reproduce social, geographic, and editorial biases present in its pretrained parameters and news data.
• Always verify generated summaries against the original article before publication or high-stakes use.

License

XL-Sum is distributed under CC BY-NC-SA 4.0. This model card therefore declares the same license for the fine-tuned checkpoint. Users are responsible for checking the terms of the base model and dataset and for ensuring that their intended use is permitted. In particular, the dataset license restricts commercial use.

Citation

If you use the model, cite the XL-Sum dataset and mBART-50:

• XL-Sum: arXiv:2106.13822
• mBART-50: arXiv:2008.00401

@inproceedings{hasan-etal-2021-xl,
  title     = {{XL-Sum}: Large-Scale Multilingual Abstractive Summarization for 44 Languages},
  author    = {Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Samin, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat},
  booktitle = {Findings of ACL-IJCNLP 2021},
  year      = {2021},
  pages     = {4693--4703},
  eprint    = {2106.13822},
  archivePrefix = {arXiv}
}

@article{tang2020multilingual,
  title   = {Multilingual Translation with Extensible Multilingual Pretraining and Finetuning},
  author  = {Tang, Yuqing and Tran, Chau and Li, Xian and Chen, Peng-Jen and Goyal, Naman and Chaudhary, Vishrav and Gu, Jiatao and Fan, Angela},
  journal = {arXiv preprint arXiv:2008.00401},
  year    = {2020},
  eprint  = {2008.00401},
  archivePrefix = {arXiv}
}

Model Card Author

Mert Samet Kayacıoğlu