MiniCPM5-2B-catalan-chat
A conversational Catalan chat model: LoRA fine-tune of openbmb/MiniCPM5-2B on the full projecte-aina/InstruCAT instruction dataset (165,100 samples, 11 task categories, ~43M tokens, 1 epoch).
Model Details
- Base model: openbmb/MiniCPM5-2B (2.5B params, Llama architecture, Apache-2.0)
- Fine-tuned on: projecte-aina/InstruCAT — Catalan instructions converted from caBreu, CatalanQA, CaWikiTC, ceil, CoqCat, GuiaCat, IntoxiCat, NLUCat, Parafraseja, PAWS-ca, sts-ca, teca and WikiCat
- Language: Catalan (ca-ES)
- Task: conversational instruction following
- License: base model Apache-2.0; training data CC-BY-NC-ND-4.0 (non-commercial) — check the InstruCAT license terms before commercial use
Training
| Setting | Value |
|---|---|
| Method | LoRA (r=32, α=64, dropout 0.05, all attention+MLP projections) |
| Trainer | TRL SFTTrainer v1.12.0, transformers 5.16.1, peft 0.20.0 |
| Prompt format | MiniCPM5 chat template; system prompt "Ets un assistent conversacional que respon sempre en català."; completion-only loss on the assistant turn |
| Sequence length | 2048, packed |
| Effective batch | 32 (4 × grad-accum 8) |
| LR / schedule | 2e-4, cosine, 200 warmup steps |
| Steps / hardware | 711 steps (~19h) on 1× A10G 24GB, bf16 + gradient checkpointing |
Observed training curve (completion-only loss, every 10 steps): 2.39 → 1.48 (step 20) → 1.04 (step 30) → 0.45 (step ~70) → ~0.35–0.40 at the end. Mean token accuracy reached ~0.91–0.92 by the second half of the run. Eval on 1,000 held-out validation samples ran at step 500 without errors.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"luispoveda93/MiniCPM5-2B-catalan-chat",
dtype="bfloat16",
)
tok = AutoTokenizer.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat")
messages = [{"role": "system", "content": "Ets un assistent conversacional que respon sempre en català."},
{"role": "user", "content": "Explica'm què és la Sardana."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
The LoRA adapter alone is at luispoveda93/MiniCPM5-2B-catalan-chat-lora.
Limitations
- Trained on single-turn instruction data; no multi-turn Catalan conversations were used.
- InstruCAT is task-oriented (QA, paraphrase, NER, classification, summarization); general chit-chat behavior comes from the base model.
- Non-commercial data license applies to the fine-tuning dataset.
Citation
Please cite the InstruCAT dataset and Projecte AINA if you use this model:
- Dataset: projecte-aina/InstruCAT — Language Technologies Unit (langtech@bsc.es) at BSC, funded within project ILENIA (ref. 2022/TL22/00215337).