T5 Small fine-tuned on BillSum for Abstractive Summarization
Model Description
This model is a fine-tuned version of google-t5/t5-small for abstractive summarization of legislative text.
It was fine-tuned on the California state bill subset of the BillSum dataset and evaluated with ROUGE.
The model was created as a practical exercise while following the Hugging Face LLM Course.
- Developed by: Driw0x
- Model type: T5 encoder-decoder Transformer
- Language: English
- Base model:
google-t5/t5-small - Task: Abstractive summarization / sequence-to-sequence generation
Training and Evaluation Data
The model was fine-tuned on:
FiscalNote/billsum
The notebook loads the California state bill subset:
billsum = load_dataset("FiscalNote/billsum", split="ca_test")
billsum = billsum.train_test_split(test_size=0.2)
This creates an 80/20 train/evaluation split from ca_test.
Because this generated evaluation split is used during model development, it should not be considered a completely untouched final benchmark.
Preprocessing
T5 inputs are prefixed with:
summarize: <document>
The preprocessing configuration is:
- maximum input length:
1024tokens; - maximum target summary length:
128tokens; - truncation enabled;
- sequence-to-sequence padding with
DataCollatorForSeq2Seq.
Training Procedure
The model was fine-tuned with Seq2SeqTrainer.
Training Hyperparameters
| Hyperparameter | Value |
|---|---|
| Learning rate | 2e-5 |
| Train batch size | 16 |
| Evaluation batch size | 16 |
| Number of epochs | 4 |
| Weight decay | 0.01 |
| Seed | 42 |
| Evaluation strategy | epoch |
| LR scheduler | linear |
| Mixed precision | FP16 / Native AMP |
| Predict with generate | True |
| Save total limit | 3 |
| Optimizer | AdamW (ADAMW_TORCH_FUSED) |
Evaluation uses ROUGE with stemming enabled.
Training Results
| Epoch | Validation Loss | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum | Generated Length |
|---|---|---|---|---|---|---|
| 1 | 2.6039 | 0.1365 | 0.0440 | 0.1119 | 0.1117 | 19.0 |
| 2 | 2.5241 | 0.1466 | 0.0524 | 0.1196 | 0.1196 | 19.0 |
| 3 | 2.4884 | 0.1535 | 0.0597 | 0.1271 | 0.1271 | 19.0 |
| 4 | 2.4781 | 0.1558 | 0.0618 | 0.1288 | 0.1288 | 19.0 |
Final reported evaluation results:
- Validation loss:
2.4781 - ROUGE-1:
0.1558 - ROUGE-2:
0.0618 - ROUGE-L:
0.1288 - ROUGE-Lsum:
0.1288 - Mean generated length:
19.0
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
model_id = "Driw0x/my_awesome_billsum_model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
text = """summarize: The Inflation Reduction Act lowers prescription drug costs,
health care costs, and energy costs. It also includes major climate and energy
investments and measures intended to reduce the federal deficit."""
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Uses
This model is primarily intended for:
- learning abstractive summarization with Transformers;
- experimenting with T5 fine-tuning;
- reproducing a Hugging Face summarization workflow;
- educational experiments on legislative text.
Limitations
Important limitations include:
- training used only the California state bill subset selected by the course workflow;
- the model is based on the relatively small
t5-smallcheckpoint; - the reported ROUGE scores remain limited;
- generated summaries may omit important details or introduce unsupported information;
- performance may degrade on documents that differ strongly from legislative text;
- biases and errors inherited from the base model and training data may remain.
This model was created as a course exercise and has not been validated for production, legal analysis, or other high-stakes use.
Framework Versions
- Transformers 5.17.0
- PyTorch 2.11.0+cu130
- Datasets 4.8.5
- Tokenizers 0.23.2
Training Source
The complete training procedure is available in:
Driw0x/hf-ai-courses
Notebook:
llm-course/1-transformer-models/notebooks/summarization.ipynb