long-t5-tglobal-base-sci-simplify

Exploring how well long-document models trained on "lay summaries" of scientific papers generalize.

A lay summary is a summary of a research paper or scientific study that is written in plain language, without the use of technical jargon, and is designed to be easily understood by non-experts.

Model description

This model is a fine-tuned version of google/long-t5-tglobal-base on the pszemraj/scientific_lay_summarisation-plos-norm dataset for two epochs.

• The variant trained on the ELIFE subset can be found here

Usage

It's recommended to use this model with beam search decoding. If you are interested, you can also use the textsum util repo to have most of this abstracted for you:

Install with pip:

pip install -U textsum

Use in python:

from textsum.summarize import Summarizer

summarizer = Summarizer('pszemraj/long-t5-tglobal-base-sci-simplify')
text = "put the text you don't want to read here"
summary = summarizer.summarize_string(text)
print(summary)

Intended uses & limitations

• Ability to generalize outside of the dataset domain (pubmed/bioscience type papers) has to be evaluated.

Training procedure

Eval results

It achieves the following results on the evaluation set:

• Loss: 1.6778
• Rouge1: 49.1475
• Rouge2: 18.9281
• Rougel: 26.9893
• Rougelsum: 45.0973
• Gen Len: 399.4125

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0004
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.01
• num_epochs: 2.0

Training results