T5 Small fine-tuned on OPUS Books for English-to-French Translation
Model Description
This model is a fine-tuned version of google-t5/t5-small for English-to-French translation.
It was fine-tuned on the English-French configuration of the OPUS Books dataset and evaluated with SacreBLEU.
The model was created as a practical exercise while following the Hugging Face LLM Course.
- Developed by: Driw0x
- Model type: T5 encoder-decoder Transformer
- Languages: English → French
- Base model:
google-t5/t5-small - Task: Machine translation / sequence-to-sequence generation
Training and Evaluation Data
The model was fine-tuned on:
Helsinki-NLP/opus_books
Configuration:
en-fr
The original training split is divided into a new 80/20 train/evaluation split:
books = load_dataset("Helsinki-NLP/opus_books", "en-fr")
books = books["train"].train_test_split(test_size=0.2)
Because the generated evaluation split is used during model development, it should not be considered a completely untouched final benchmark.
Preprocessing
T5 is prompted explicitly for the translation task:
translate English to French: <text>
The preprocessing configuration is:
- source language: English (
en); - target language: French (
fr); - maximum tokenized length:
128; - truncation enabled;
- dynamic sequence-to-sequence padding with
DataCollatorForSeq2Seq.
Training Procedure
The model was fine-tuned with Seq2SeqTrainer.
Training Hyperparameters
| Hyperparameter | Value |
|---|---|
| Learning rate | 2e-5 |
| Train batch size | 16 |
| Evaluation batch size | 16 |
| Number of epochs | 2 |
| Weight decay | 0.01 |
| Seed | 42 |
| Evaluation strategy | epoch |
| LR scheduler | linear |
| Mixed precision | FP16 / Native AMP |
| Predict with generate | True |
| Save total limit | 3 |
| Optimizer | AdamW (ADAMW_TORCH_FUSED) |
Evaluation uses SacreBLEU on generated translations.
Training Results
| Epoch | Training Loss | Validation Loss | BLEU | Generated Length |
|---|---|---|---|---|
| 1 | 1.7705 | 1.5670 | 5.9461 | 17.5581 |
| 2 | 1.7483 | 1.5517 | 6.0284 | 17.5540 |
Final reported evaluation results:
- Validation loss:
1.5517 - SacreBLEU:
6.0284 - Mean generated length:
17.554
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
model_id = "Driw0x/my_awesome_opus_books_model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
text = "translate English to French: Legumes share resources with nitrogen-fixing bacteria."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=40,
do_sample=True,
top_k=30,
top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Uses
This model is primarily intended for:
- learning sequence-to-sequence translation with Transformers;
- experimenting with T5 fine-tuning;
- reproducing a Hugging Face translation workflow;
- educational English-to-French translation experiments.
Limitations
Important limitations include:
- the model is based on the relatively small
t5-smallcheckpoint; - the reported BLEU score is modest and the model should not be treated as a production translation system;
- OPUS Books is a literary-domain dataset and may not generalize well to conversational, technical, legal, or specialized text;
- sampled generation can produce different outputs across runs;
- biases and errors inherited from the base model and training corpus may remain.
This model was created as a course exercise and has not been validated for production or high-stakes translation.
Framework Versions
- Transformers 5.17.0
- PyTorch 2.11.0+cu130
- Datasets 4.8.5
- Tokenizers 0.23.2
Training Source
The complete training procedure is available in:
Driw0x/hf-ai-courses
Notebook:
llm-course/1-transformer-models/notebooks/translation.ipynb