Qwen2.5-0.5B ALMA (English to Spanish translation)
A small decoder-only translation model produced by fine-tuning Qwen/Qwen2.5-0.5B using the two-stage ALMA recipe:
- Stage 1 — continued monolingual pretraining on Spanish text (
<your-username>/qwen2.5-0.5b-mono-es) - Stage 2 (this checkpoint) — supervised fine-tuning on parallel English to Spanish sentence pairs, starting from the Stage 1 checkpoint
Training details
- Base model: Qwen/Qwen2.5-0.5B, via the Stage 1 continued-pretraining checkpoint
- Data: Helsinki-NLP/tatoeba_mt (eng-spa), validation split, ~197k sentence pairs (the test split was held out entirely for evaluation and never seen in training)
- Format:
Translate this from English to Spanish:\nEnglish: {src}\nSpanish: {tgt}, with the loss masked (-100) over the prompt/source span so only the Spanish completion is supervised - Optimizer / schedule: AdamW, lr=2e-5, cosine schedule, 3% warmup, 1 epoch
- Precision: bf16
- Effective batch size: 32 (
per_device_train_batch_size=8xgradient_accumulation_steps=4), max sequence length 256
Evaluation
Zero-shot BLEU on 100 held-out Helsinki-NLP/tatoeba_mt (eng-spa) test sentences, disjoint from all training data:
| Model | Prompting | BLEU |
|---|---|---|
| Qwen2.5-0.5B (pretrained) | 5-shot | 30.54 |
| Qwen2.5-0.5B + mono-es pretrain only | 5-shot | 29.59 |
| This model (full ALMA) | 0-shot | 44.91 |
Nearly all of the improvement over the base model comes from this parallel fine-tuning stage rather than the monolingual continued-pretraining step.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("<your-username>/qwen2.5-0.5b-alma-es")
model = AutoModelForCausalLM.from_pretrained("<your-username>/qwen2.5-0.5b-alma-es")
prompt = "Translate this from English to Spanish:\nEnglish: Good morning.\nSpanish:"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Limitations
Trained on short, colloquial Tatoeba-style sentences; not evaluated on longer-form or domain-specific text. At 0.5B parameters it still makes occasional lexical errors (e.g. mistranslating numbers) despite the aggregate BLEU improvement over the baseline.