scihigh2026-subtask1-bart-large-cnn
Fine-tuned facebook/bart-large-cnn
for SciHigh-2026 Subtask 1: Research Highlight Generation — generating
3-5 short, telegraphic highlight sentences (~50-55 words total) from a
scientific paper's abstract.
Task
Given a paper's abstract, generate a small set of highlight sentences that summarize the paper's key contributions/findings in a compressed, telegraphic style (not a paraphrase-style abstractive summary). Ranked by ROUGE-L F1 (primary metric), with ROUGE-1/2, METEOR, and BERTScore also reported.
Baseline: the FIRE-2025 shared-task's winning submission (fine-tuned Pegasus-large + NER features, 10 epochs) scored 23.45% ROUGE-L F1.
Training data
Fine-tuned on the MixSub-SciHigh dataset, an expanded pool of
15,960 (Abstract, Highlights) pairs: the official 2026 task's
10,000-row training split, plus 5,960 additional real training pairs
recovered from the dataset's original source release (leakage-checked by
exact Abstract-text match against the official validation/test splits before
being used for training — zero overlap confirmed). Validated against the
official held-out validation split (1,985 rows), which was not used for
training, only for per-epoch checkpoint selection.
Training recipe
| Setting | Value |
|---|---|
| Backbone | facebook/bart-large-cnn |
| Learning rate | 2e-5 (Adafactor optimizer) |
| Batch size | 2 (per device) |
| Warmup / label smoothing | none |
| Max input / output length | 512 / 100 tokens |
| Beam width | 4 |
| Epoch ceiling | 4, with load_best_model_at_end=True + EarlyStoppingCallback(patience=2) on validation ROUGE-L |
| Decode settings | no_repeat_ngram_size=3 (explicit); repetition_penalty and min_length left at bart-large-cnn's own shipped defaults (1.0 / 56) |
Hyperparameters mirror the FIRE-2025 baseline recipe unchanged (a hyperparameter sweep over learning rate, warmup, and label smoothing found every variant flat-to-worse on top of this backbone). The epoch ceiling is a safety bound, not a fixed schedule — early stopping determines the actual stopping point.
In the training run that produced this checkpoint, validation ROUGE-L peaked after epoch 1 and did not improve in epochs 2 or 3; early stopping halted training after epoch 3 (before reaching the 4-epoch ceiling), and the epoch-1 checkpoint (this one) was restored as the final model.
Evaluation results
Evaluated on the full official validation split (1,985 rows):
| Metric | Score | vs. FIRE-2025 baseline (23.45% ROUGE-L) |
|---|---|---|
| ROUGE-1 | 36.22% | — |
| ROUGE-2 | 13.17% | — |
| ROUGE-L | 24.37% | +0.92 pts absolute (+3.9% relative) |
| ROUGE-Lsum | 24.36% | — |
| METEOR | 32.48% | — |
| BERTScore F1 | 87.49% | — |
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "Yakk99/scihigh2026-subtask1-bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
abstract = "..." # a scientific paper's abstract
inputs = tokenizer(abstract, max_length=512, truncation=True, return_tensors="pt")
output = model.generate(**inputs, max_length=100, num_beams=4)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])
max_length=100 is set explicitly to match the recipe's output-length
budget — the checkpoint's own generation_config already carries this value,
so a plain model.generate(**inputs) call (with no extra arguments) also
reproduces the reported results.
Files
train_final.py/train_final.ipynb— the standalone training script and a companion notebook documenting this exact recipe end-to-end (data, model, training configuration, decode settings, and the real reported results above).
Limitations
- Abstracts in the source dataset are frequently truncated mid-sentence (a scraped-preview artifact in the original data release), which imposes a hard ceiling on achievable ROUGE-L for the affected subset — some reference highlights cite facts genuinely absent from the (truncated) input.
- This model was selected and evaluated on ROUGE-L against a single validation split; it has not been evaluated for factual accuracy/ hallucination beyond standard n-gram/embedding-based metrics.