Text-to-Bullets T5

A fine-tuned T5 encoder-decoder model for converting English prose into concise bullet points.

Model Summary

The model was fine-tuned specifically for source-to-target text transformation.

Unlike a decoder-only model, T5 separates the task into:

Source text
   ↓
Encoder
   ↓
Source representation
   ↓
Decoder
   ↓
Bullet points

This architecture is well suited to summarization-style tasks because the decoder generates output while cross-attending to the encoded source.

Training Results

The model was fine-tuned for seven epochs.

Training observations

The largest improvement happened between epochs 1 and 2.

• ROUGE-L increased from 0.7770 → 0.9493
• Mean bullet-count error decreased from 3.36 → 0.45
• Performance largely plateaued around epochs 5–6
• Bullet-format compliance remained 100% in the reported validation evaluation

These are training-time validation results and should not be directly compared with later held-out evaluations unless the same split and generation settings are used.

Later 500-Example Evaluation

A later FP32 evaluation on 500 examples produced:

Comparison with Qwen3 0.6B

The T5 model is roughly an order of magnitude smaller by parameter count.

In the reported evaluations, Qwen retained the higher ROUGE-1 score, while the fine-tuned T5 produced higher ROUGE-2 and ROUGE-L.

These measurements were collected at different stages of the experiment, so they should be treated as experimental context rather than a perfectly controlled head-to-head benchmark.

The key takeaway is that a much smaller task-specific encoder-decoder model can be highly competitive with a larger general-purpose decoder model on a narrow text-transformation task.

Usage

Install dependencies:

pip install torch transformers sentencepiece

Load and run the model:

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

MODEL_ID = "JayShah07/falconai-text-bullet-t5"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).eval()

text = """
The Solar System consists of the Sun and the objects gravitationally bound
to it, including eight planets, dwarf planets, moons, asteroids and comets.
It formed approximately 4.6 billion years ago. Mercury, Venus, Earth and
Mars are terrestrial planets, while Jupiter and Saturn are gas giants and
Uranus and Neptune are ice giants.
""".strip()

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        num_beams=1,
    )

output = tokenizer.decode(
    output_ids[0],
    skip_special_tokens=True,
)

print(output)

Example Output Style

- The Solar System contains the Sun, eight planets, dwarf planets, moons, asteroids and comets.
- It formed approximately 4.6 billion years ago.
- Mercury, Venus, Earth and Mars are terrestrial planets.
- Jupiter and Saturn are gas giants, while Uranus and Neptune are ice giants.

Exact wording and bullet segmentation may vary.

Intended Use

This model is intended for:

• converting paragraphs into bullet points
• restructuring reports and articles
• summarization-style text transformation
• extracting key information from English prose

Limitations

The model may omit details from dense text, merge related facts, or produce imperfect summaries on text outside its training distribution.

Generated outputs should be reviewed before high-stakes use.

Author

Developed by Jay Shah as an experiment in task-specific encoder-decoder fine-tuning.