thealper2/t5-small-commitbench
google-t5/t5-small fine-tuned on Maxscha/commitbench for
commit message generation: given a git diff, generate the commit message describing it.
Task format
Text-to-text. The input is a task prefix followed by the raw git diff, the target is the commit message.
generate commit message: <git diff>
Training data
Maxscha/commitbench official splits, used unchanged:
| Split | Examples in split | Examples used |
|---|---|---|
| train | 1,165,213 | 500,000 |
| validation | 249,689 | 2,000 |
| test | 249,688 | not used for training |
Languages covered by the dataset: Python, JavaScript, PHP, Ruby, Java, Go.
Training configuration
| Setting | Value |
|---|---|
| Base model | google-t5/t5-small |
| Parameters | 60.5M |
| Max source length | 512 tokens |
| Max target length | 64 tokens |
| Per-device batch size | 32 |
| Gradient accumulation | 1 |
| Effective batch size | 32 |
| Learning rate | 3e-05 |
| LR schedule | linear |
| Warmup ratio | 0.05 |
| Weight decay | 0.01 |
| Epochs | 2.0 |
| Label smoothing | 0.0 |
| Gradient clipping | 1.0 |
| Mixed precision | bf16 |
| Seed | 42 |
| Optimizer | AdamW |
| Training time | 1.219 h |
| Hardware | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |
Truncation at these limits (measured on a 50k sample with the T5 tokenizer):
- 0.7% of the diffs exceed 512 source tokens.
- 4.47% of the commit messages exceed 64 target tokens.
Results
- Final training loss: 3.5762
- Best validation loss: 3.2414
Test split (20,000 examples), beam search with num_beams=4:
| Metric | Value |
|---|---|
| rouge1 | 19.31 |
| rouge2 | 4.668 |
| rougeL | 17.42 |
| rougeLsum | 17.42 |
| bleu | 2.148 |
| exact_match | 0.04 |
| gen_len_words_mean | 5.005 |
| ref_len_words_mean | 11.27 |
Per programming language:
| Language | n | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | Exact match |
|---|---|---|---|---|---|---|
| Python | 5,722 | 21.20 | 6.05 | 19.29 | 2.73 | 0.04 |
| JavaScript | 4,468 | 18.86 | 4.05 | 17.07 | 2.01 | 0.02 |
| PHP | 3,489 | 17.04 | 3.46 | 15.29 | 1.64 | 0.09 |
| Ruby | 2,808 | 22.08 | 5.79 | 19.65 | 2.44 | 0.04 |
| Java | 1,799 | 15.19 | 2.61 | 13.58 | 1.02 | 0.06 |
| Go | 1,714 | 18.65 | 4.45 | 16.75 | 2.11 | 0.00 |
ROUGE and BLEU are lexical-overlap metrics. They do not fully capture whether a commit message describes a change correctly, and generic messages can score well.
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "thealper2/t5-small-commitbench"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
diff = open("change.patch").read()
inputs = tokenizer(
"generate commit message: " + diff,
max_length=512,
truncation=True,
return_tensors="pt",
)
output = model.generate(
**inputs,
num_beams=4,
max_new_tokens=64,
length_penalty=1.0,
no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Default generation settings: num_beams=4, max_new_tokens=64,
min_new_tokens=0, length_penalty=1.0,
no_repeat_ngram_size=3, do_sample=False (deterministic).
Limitations
- CommitBench replaces identifying literals with placeholder tokens: every diff contains
<HASH>instead of commit hashes, and 26.5% of the reference messages contain<I>(numbers),<URL>or<EMAIL>. The model therefore also generates these tokens, e.g.Bumped version to <I>. - The T5 sentencepiece vocabulary does not cover every character used in source code (curly braces, backslashes, angle brackets), so about 2.35% of the input tokens become
<unk>. This limits how precisely the model can read a diff. - Diffs longer than 512 tokens are truncated; the tail of the change is not visible to the model.
- CommitBench splits are random over commits, not over repositories: 98.6% of the test examples
come from repositories that also appear in the training split. No
(diff, message)pair is shared across splits, but the reported scores partly reflect familiarity with a project's commit style rather than generalization to unseen code. - The dataset is English-only and covers six languages; behaviour on other languages or on very large multi-file changes is untested.
- CommitBench is released under CC BY-NC 4.0, which restricts commercial use of the data.
Reproducibility
- python:
3.12.3 - torch:
2.11.0+cu128 - transformers:
5.17.0 - datasets:
4.3.0 - tokenizers:
0.23.2 - seed:
42