An mT5-base model on the pnSummary dataset to summarize articles.
Eval results
The following table summarizes the ROUGE scores obtained by the model for the validation set.
+-----------+------+-----------+--------+-----------+
| Score | Type | Precision | Recall | F-Measure |
+-----------+------+-----------+--------+-----------+
| rouge1 | low | 48.60 | 45.25 | 45.47 |
| rouge1 | mid | 49.22 | 45.79 | 46.02 |
| rouge1 | high | 49.80 | 46.36 | 46.54 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rouge2 | low | 29.46 | 27.33 | 27.56 |
| rouge2 | mid | 30.08 | 27.93 | 28.12 |
| rouge2 | high | 30.69 | 28.52 | 28.71 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeL | low | 42.20 | 39.26 | 39.51 |
| rougeL | mid | 42.80 | 39.86 | 40.07 |
| rougeL | high | 43.43 | 40.41 | 40.62 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeLsum | low | 42.15 | 39.26 | 39.48 |
| rougeLsum | mid | 42.82 | 39.87 | 40.09 |
| rougeLsum | high | 43.39 | 40.43 | 40.61 |
+-----------+------+-----------+--------+-----------+
Test results
The following table summarizes the ROUGE scores obtained by the model for the test set.
+-----------+------+-----------+--------+-----------+
| Score | Type | Precision | Recall | F-Measure |
+-----------+------+-----------+--------+-----------+
| rouge1 | low | 48.83 | 44.99 | 45.41 |
| rouge1 | mid | 49.39 | 45.52 | 45.91 |
| rouge1 | high | 50.02 | 46.07 | 46.48 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rouge2 | low | 29.27 | 26.90 | 27.23 |
| rouge2 | mid | 29.97 | 27.52 | 27.84 |
| rouge2 | high | 30.62 | 28.15 | 28.46 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeL | low | 42.30 | 38.94 | 39.34 |
| rougeL | mid | 42.85 | 39.54 | 39.90 |
| rougeL | high | 43.51 | 40.12 | 40.49 |
| | | | | |
| *** | *** | *** | *** | *** |
| | | | | |
| rougeLsum | low | 42.27 | 38.94 | 39.35 |
| rougeLsum | mid | 42.86 | 39.53 | 39.90 |
| rougeLsum | high | 43.42 | 40.12 | 40.45 |
+-----------+------+-----------+--------+-----------+