flan-t5-small-sentence-compression
Full fine-tune of google/flan-t5-small for English sentence compression, trained on sentence-transformers/sentence-compression as a sequence-to-sequence task (text → simplified).
Model
| Parameter | Value |
|---|---|
| Base model | google/flan-t5-small |
| Architecture | T5 encoder-decoder (T5ForConditionalGeneration) |
| Parameters | 76,961,152 |
| Fine-tuning | Full fine-tuning (no adapters, no quantization) |
| Input format | compress sentence: {text} |
| Max source length | 256 tokens |
| Max target length | 64 tokens |
Data
| Parameter | Value |
|---|---|
| Dataset | sentence-transformers/sentence-compression (train split only) |
| Mapping | text → simplified |
| Original examples | 180,000 |
| Removed (missing / empty / duplicate pairs) | 0 / 0 / 0 |
| Train examples | 171,000 |
| Validation examples | 9,000 |
| Split | train_test_split(test_size=0.05, seed=42) |
| Train/validation source overlap | 0 |
| Source token length (mean / p99 / max) | 38.8 / 81 / 1212 |
| Target token length (mean / p99 / max) | 11.2 / 23 / 43 |
| Truncated sources / targets | 0.04% / 0.00% |
The dataset has no dedicated test split; all reported metrics are on the held-out validation split.
Training
| Parameter | Value |
|---|---|
| Epochs | 3 |
| Optimizer | adamw_torch_fused |
| Learning rate | 0.0003 |
| LR schedule | linear, warmup ratio 0.05 |
| Weight decay | 0.01 |
| Per-device batch size | 32 |
| Gradient accumulation | 1 |
| Effective batch size | 32 |
| Optimizer steps | 16,032 |
| Precision | BF16 mixed precision |
| Seed | 42 |
| Best checkpoint selection | eval_loss = 0.7334 |
| Final train loss | 0.8378 |
| GPU | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |
| Peak VRAM (reserved) | 6.52 GB |
| Training time | 0.72 h |
| Throughput | 199.2 samples/s |
| Software | PyTorch 2.11.0+cu128, CUDA 12.8, Transformers 5.17.0 |
Evaluation
Validation split, 9,000 examples. Generation: num_beams=4, do_sample=False, max_new_tokens=64, length_penalty=1.0, no_repeat_ngram_size=3. ROUGE is the mean per-example F1 (0-100) with Porter stemming.
| Metric | Value |
|---|---|
| ROUGE-1 | 77.68 |
| ROUGE-2 | 63.47 |
| ROUGE-L | 77.35 |
| Avg. source length | 24.89 words / 34.94 tokens |
| Avg. reference length | 6.96 words / 10.11 tokens |
| Avg. generated length | 7.28 words / 10.44 tokens |
| Compression ratio (generated/source, words) | 0.325 |
| Compression ratio (generated/source, tokens) | 0.330 |
| Shorter than source | 99.72% |
| Approximately same length (±10% words) | 0.28% |
| Longer than source | 0.00% |
| Exact copies of source | 0.03% |
Base model comparison
Same 1,046 validation examples, same input format and generation parameters.
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | Avg. generated words | Compression ratio (words) |
|---|---|---|---|---|---|
Base google/flan-t5-small | 47.11 | 29.83 | 46.44 | 21.14 | 0.874 |
| Fine-tuned | 78.59 | 64.59 | 78.29 | 7.20 | 0.319 |
Decoding settings
1,000 validation examples.
| Generation parameters | ROUGE-1 | ROUGE-2 | ROUGE-L | Avg. generated words |
|---|---|---|---|---|
do_sample=False, max_new_tokens=64, num_beams=1 | 78.07 | 64.07 | 77.77 | 6.91 |
do_sample=False, max_new_tokens=64, num_beams=4 | 78.60 | 64.56 | 78.30 | 7.19 |
do_sample=False, max_new_tokens=64, num_beams=4, no_repeat_ngram_size=3 | 78.60 | 64.56 | 78.30 | 7.19 |
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "thealper2/flan-t5-small-sentence-compression"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
text = "Major League Baseball Commissioner Bud Selig will be speaking at St. Norbert College next month."
inputs = tokenizer("compress sentence: " + text, return_tensors="pt", truncation=True, max_length=256)
outputs = model.generate(**inputs, num_beams=4, do_sample=False, max_new_tokens=64, length_penalty=1.0, no_repeat_ngram_size=3)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The compress sentence: prefix is required; the model was trained only with this input format.
Limitations
- English only.
- References in the training data are headline-style compressions of news sentences; outputs follow that style (average 7.3 words).
- Inputs longer than 256 tokens (including the prefix) are truncated.
- ROUGE measures lexical overlap with a single reference; it does not measure factual consistency.