English → Welsh Neural Machine Translation
A Transformer-based sequence-to-sequence model for English to Welsh translation, implemented from scratch in PyTorch. Full training code available at github.com/mdpead/en-cy-translation.
Usage
from transformers import pipeline
pipe = pipeline("translation", model="mdpead/en-cy-translation")
pipe("Hello, how are you?")
Or with the model and tokenizer directly:
from src.hf_wrapper import EnCyForTranslation
from transformers import PreTrainedTokenizerFast
model = EnCyForTranslation.from_pretrained("mdpead/en-cy-translation")
tokenizer = PreTrainedTokenizerFast.from_pretrained("mdpead/en-cy-translation")
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
output_ids = model.generate(**inputs, max_length=256)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Architecture
| Component | Detail |
|---|---|
| Model | Encoder-Decoder Transformer |
Embedding dim (d_model) | 512 |
| Attention heads | 8 |
| Encoder / Decoder layers | 6 / 6 |
Feed-forward dim (d_ff) | 2048 |
| Vocabulary size | 16,000 |
| Max sequence length | 256 tokens |
| Tokenizer | WordPiece (shared bilingual vocabulary) |
Training uses mixed-precision (AMP), gradient accumulation, and a warmup inverse square-root learning rate schedule.
Training
- Dataset: techiaith/cardiff-university-tm-en-cy (~1.3M sentence pairs)
- Steps: 50,000
- Effective batch size: 25,000 tokens
- Optimiser: AdamW (β₁=0.9, β₂=0.98, ε=1e-9)
- Learning rate: 1e-3 with 2,000 warmup steps, inverse square root decay
Benchmark
Evaluated with beam search (beam size 4) against three publicly available EN→CY models across three datasets:
- FLORES+:
openlanguagedata/flores_plusdevtest split (1012 sentences, Wikipedia text) - Cardiff:
techiaith/cardiff-university-tm-en-cy10% held-out test split (1000 sentences, institutional text) - Tatoeba:
agentlans/tatoeba-english-translationsWelsh subset (1613 sentences, casual/short)
FLORES+ (out-of-distribution, Wikipedia)
| Model | Params | BLEU | spBLEU | chrF | chrF++ |
|---|---|---|---|---|---|
| en-cy-translation | ~50M | 46.29 | 51.89 | 68.39 | 66.38 |
| NLLB-200 (distilled) | 600M | 36.14 | 37.57 | 58.61 | 56.55 |
| Opus-MT | 74M | 13.94 | 13.74 | 33.82 | 32.25 |
| Small-100 | 330M | 1.96 | 2.86 | 18.83 | 16.14 |
Cardiff University TM (in-distribution, institutional)
| Model | BLEU | spBLEU | chrF | chrF++ |
|---|---|---|---|---|
| en-cy-translation | 57.49 | 63.87 | 76.77 | 74.64 |
| NLLB-200 (distilled) | 37.10 | 39.51 | 61.36 | 58.69 |
| Opus-MT | 11.05 | 10.69 | 31.19 | 29.17 |
| Small-100 | 1.54 | 2.02 | 17.25 | 14.52 |
Tatoeba (casual, short sentences)
| Model | BLEU | spBLEU | chrF | chrF++ |
|---|---|---|---|---|
| en-cy-translation | 47.37 | 50.21 | 66.57 | 64.22 |
| NLLB-200 (distilled) | 39.12 | 39.15 | 58.21 | 56.17 |
| Opus-MT | 29.86 | 30.11 | 48.25 | 46.72 |
| Small-100 | 0.52 | 0.75 | 11.63 | 10.84 |
Setup
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python scripts/train.py --config base
Checkpoints are saved to runs/<run-name>/checkpoints/ every checkpoint_steps steps. Training resumes automatically from the latest checkpoint.
Project Structure
├── configs/ # YAML training configs
├── scripts/
│ ├── train.py # Training entry point
│ └── push_to_hub.py
└── src/
├── model.py # Transformer implementation
├── tokenizer.py # WordPiece tokenizer
├── datasets.py # Dataset loading
├── dataloader.py # Token-bucketed DataLoader
├── train.py # Training loop and checkpointing
├── generation.py # Autoregressive decoding
└── hf_wrapper.py # HuggingFace PreTrainedModel wrapper
License
CC BY 4.0 — derived from the Cardiff University Translation Memory dataset, also licensed CC BY 4.0. Attribution to Cardiff University Language Technologies Unit.