t5-efficient-base-grammar-correction
Full fine-tune (all parameters, no adapters) of google/t5-efficient-base for English sentence-level grammatical error correction on agentlans/grammar-correction.
Model
| Property | Value |
|---|---|
| Architecture | T5 encoder-decoder (12 + 12 layers) |
| d_model / d_ff / heads | 768 / 3072 / 12 |
| Parameters | 222,903,552 |
| Vocabulary | SentencePiece, 32,128 embeddings |
| Input format | grammar correction: {sentence} |
| Max source / target tokens | 256 / 256 |
Training data
| Split | Examples | Use |
|---|---|---|
| train | 100,000 | training |
| validation | 2,000 | per-epoch model selection (seeded subset) |
| validation | 25,000 | final evaluation below |
- Token length (train, input with prefix): mean 38.23, p99 129; truncated at 256/256: 0.10% sources, 0.03% targets.
- Duplicate pairs: 0 (train), 0 (validation); validation inputs present in train: 0.
- Pairs with input == output: 0.
- Targets containing characters outside the SentencePiece vocabulary (
<unk>): 1.31%. - Text is used as-is (no normalization of case, punctuation or contractions).
Training procedure
| Hyperparameter | Value |
|---|---|
| Optimizer | adamw_torch_fused |
| Learning rate | 0.0001 |
| LR schedule | linear, warmup ratio 0.05 |
| Epochs | 5.00 |
| Batch size (per device x accumulation) | 16 x 2 |
| Effective batch size | 32 |
| Optimization steps | 15,625 |
| Weight decay | 0.01 |
| Label smoothing | 0.00 |
| Max grad norm | 1.00 |
| Precision | bf16 |
| Gradient checkpointing | no |
| Seed | 42 |
| Hardware | NVIDIA GeForce RTX 5060 Ti |
| Training time | 1.90 h |
Evaluation
Checkpoint selected by eval_sari on 2,000 validation examples: epoch 5.00, step 15,625, validation loss 0.6534 (perplexity 1.922).
| Metric | Model (n=25,000) | Copy-input baseline |
|---|---|---|
| SARI | 68.58 | 58.40 |
| BLEU (sacrebleu) | 70.24 | 63.57 |
| chrF | 84.41 | 82.49 |
| Exact match (%) | 9.10 | 0.00 |
| Output identical to input (%) | 7.38 | 100.00 |
Validation loss 0.6504, perplexity 1.916. Decoding: beam search, 4 beams.
Over-correction check: 1,000 grammatical validation references fed as inputs; 24.20% were modified (chrF vs. input 98.81).
Metric notes: BLEU and chrF measure overlap with the reference, so copying the input already scores high (see baseline). SARI (Xu et al., 2016) scores kept/added/deleted n-grams relative to the source on lowercased 13a tokens, so casing edits are not reflected. Exact match is case- and punctuation-sensitive. ERRANT F0.5 was not computed.
Generation config
{
"max_new_tokens": 256,
"early_stopping": true,
"do_sample": false,
"num_beams": 4,
"pad_token_id": 0,
"eos_token_id": 1,
"decoder_start_token_id": 0
}
Usage
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "thealper2/t5-efficient-base-grammar-correction"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id).eval()
text = "She go to school yesterday."
inputs = tokenizer("grammar correction: " + text, return_tensors="pt", truncation=True, max_length=256)
with torch.inference_mode():
output_ids = model.generate(**inputs) # uses the bundled generation_config.json
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Limitations
- Training pairs are synthetic (C4_200M-derived corruptions filtered by a grammar classifier); error distribution differs from learner or native-speaker text.
- English only, sentence/short-paragraph level; inputs longer than 256 tokens are truncated.
- The training set contains no already-correct (input == output) pairs, so leaving correct text unchanged is not directly supervised; check the over-correction numbers above.
- The T5 SentencePiece vocabulary cannot represent some characters (e.g.
{,},<,~, many accented letters); they map to<unk>and are dropped from generated text. - Not evaluated on standard GEC benchmarks (CoNLL-2014, BEA-2019).