mBART XL-Sum 2

mbart-xlsum-2 is a multilingual abstractive news summarization model fine-tuned from facebook/mbart-large-50-many-to-many-mmt on 15 language subsets of csebuetnlp/xlsum. It was developed as part of a multilingual news aggregation and summarization project.

This is the project's quality-oriented second mBART-50 training variant. It uses a shorter 64-token training target, generation-based validation, ROUGE-L checkpoint selection, and alpha-smoothed language resampling to increase the visibility of lower-resource languages. In the final evaluation it achieved small gains over the first variant in ROUGE, BLEU, METEOR-lite, latency, completeness, and bigram novelty, while showing lower source coverage in the project's automatic diagnostics.

Related Project

This model was developed as part of samurAI, a multilingual news aggregation and summarization system with RSS ingestion, local transformer summarization, SQLite storage, a Flask web interface, JSON APIs, and evaluation notebooks.

Related checkpoint: mskayacioglu/mbart50-xlsum

Model Details

Languages

The fine-tuning mixture contains the following 15 languages:

Quantitative evaluation was completed for 11 of these languages: Arabic, Chinese (Simplified), English, French, Hindi, Japanese, Korean, Russian, Spanish, Turkish, and Vietnamese. Results for Indonesian, Persian, Portuguese, and Ukrainian are not reported in the final common evaluation, so performance claims should not be extrapolated to them.

Intended Use

The model is intended for:

• summarizing news articles in a supported source language;
• research on multilingual abstractive summarization;
• experiments involving low-resource language balancing;
• integration into news-processing pipelines with human review.

The model is not intended to be a factual authority, to make editorial decisions autonomously, or to summarize high-stakes medical, legal, financial, or emergency information without verification against the source.

Training Data

The checkpoint was trained on csebuetnlp/xlsum, a multilingual collection of BBC news articles paired with short reference summaries. Examples from the 15 selected subsets were annotated with their corresponding mBART language codes.

Language Resampling

To reduce domination by high-resource languages, training examples were resampled with:

p(language) proportional to n(language)^alpha, where alpha = 0.5

The total training-set size was kept approximately constant. High-resource languages were downsampled while lower-resource languages were oversampled with replacement. This improves exposure balance but can also increase repetition of examples from the smallest subsets.

Training Procedure

Gradient checkpointing and generation-based validation were enabled. Training ran on an NVIDIA A100-SXM4-40GB GPU and completed with a reported training loss of 2.1509 in approximately 12 hours 7 minutes.

The selected checkpoint reported the following internal held-out results on 7,500 examples: validation ROUGE-L 0.1860 and test ROUGE-L 0.1851. These results used the training notebook's evaluation path and are not directly interchangeable with the common final evaluation below.

Evaluation

Protocol

The final evaluation used the complete available test subsets of csebuetnlp/xlsum for 11 languages, totaling 52,219 examples per model. No per-language sample cap was applied. Scores below are macro-averaged across languages and reported on a 0-1 scale.

Articles were truncated to 512 tokens. Evaluation used batch size 8 and mixed precision on CUDA. The common mBART decoding configuration was:

max_length=90
min_length=24
num_beams=5
length_penalty=1.1
no_repeat_ngram_size=3
repetition_penalty=1.05
early_stopping=True

Turkish used a language-specific configuration with max_length=84, min_length=26, 6 beams, length penalty 1.3, 4-gram repetition blocking, and repetition penalty 1.2. The target language token was forced for every language.

Overall Results

METEOR-lite is the project's lightweight exact-token implementation and must not be interpreted as the canonical METEOR package score.

Results by Language

Supplementary Diagnostics

These diagnostics are included to describe model behavior, not as established factuality or human-quality measures.

Latency was measured in one CUDA evaluation environment and is not portable across hardware, software versions, or batch configurations. The completeness and factuality values are project-specific automatic proxies; they do not prove semantic completeness or factual correctness.

Usage

import torch
from transformers import MBart50TokenizerFast, MBartForConditionalGeneration

model_id = "mskayacioglu/mbart-xlsum-2"
language_code = "tr_TR"

tokenizer = MBart50TokenizerFast.from_pretrained(model_id)
model = MBartForConditionalGeneration.from_pretrained(model_id)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

article = "Özetlenecek Türkçe haber metni buraya yazılır."
tokenizer.src_lang = language_code

inputs = tokenizer(
    article,
    return_tensors="pt",
    max_length=512,
    truncation=True,
).to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.lang_code_to_id[language_code],
        max_length=84,
        min_length=26,
        num_beams=6,
        length_penalty=1.3,
        no_repeat_ngram_size=4,
        repetition_penalty=1.2,
        early_stopping=True,
    )

summary = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
print(summary)

For languages other than Turkish, the evaluation configuration used max_length=90, min_length=24, 5 beams, length penalty 1.1, 3-gram repetition blocking, and repetition penalty 1.05.

For languages other than Turkish, the evaluation configuration used max_length=90, min_length=24, 5 beams, length penalty 1.1, 3-gram repetition blocking, and repetition penalty 1.05.

Limitations and Risks

• The model can hallucinate facts, names, dates, quantities, or causal relationships.
• Inputs longer than 512 tokens were truncated during fine-tuning and evaluation; important information near the end of long articles may be omitted.
• XL-Sum is based on BBC news. Performance may decline on other publishers, informal writing, specialist domains, or structurally noisy web text.
• Automatic overlap metrics reward lexical similarity and do not establish factual consistency or summary usefulness.
• Evaluation coverage is uneven across languages and includes only 11 languages in the final common benchmark.
• Alpha-smoothed resampling duplicates examples from smaller language subsets and may increase memorization or language-specific overfitting.
• Higher bigram novelty indicates more reformulation, but may also increase the risk of departing from the source.
• The model may reproduce social, geographic, and editorial biases present in its pretrained parameters and news data.
• Always verify generated summaries against the original article before publication or high-stakes use.

License

XL-Sum is distributed under CC BY-NC-SA 4.0. This model card therefore declares the same license for the fine-tuned checkpoint. Users are responsible for checking the terms of the base model and dataset and for ensuring that their intended use is permitted. In particular, the dataset license restricts commercial use.

Citation

If you use the model, cite the XL-Sum dataset and mBART-50:

• XL-Sum: arXiv:2106.13822
• mBART-50: arXiv:2008.00401

@inproceedings{hasan-etal-2021-xl,
  title     = {{XL-Sum}: Large-Scale Multilingual Abstractive Summarization for 44 Languages},
  author    = {Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Samin, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat},
  booktitle = {Findings of ACL-IJCNLP 2021},
  year      = {2021},
  pages     = {4693--4703},
  eprint    = {2106.13822},
  archivePrefix = {arXiv}
}

@article{tang2020multilingual,
  title   = {Multilingual Translation with Extensible Multilingual Pretraining and Finetuning},
  author  = {Tang, Yuqing and Tran, Chau and Li, Xian and Chen, Peng-Jen and Goyal, Naman and Chaudhary, Vishrav and Gu, Jiatao and Fan, Angela},
  journal = {arXiv preprint arXiv:2008.00401},
  year    = {2020},
  eprint  = {2008.00401},
  archivePrefix = {arXiv}
}

Model Card Author

Mert Samet Kayacıoğlu