MiniCPM5-2B-catalan-chat-v2
Conversational Catalan chat model — round 2. A LoRA fine-tune of luispoveda93/MiniCPM5-2B-catalan-chat (itself a LoRA fine-tune of openbmb/MiniCPM5-2B on InstruCAT) focused on multi-turn chat behaviour.
What round 2 added
Round 1 trained on single-turn task instructions (projecte-aina/InstruCAT). Round 2 trains on 23.4K multi-turn Catalan conversations from the chat-dense sources of BSC-LT/ALIA-2606-SFT — the SFT mixture that trained ALIA-40b-instruct-2606:
| Source | Conversations | What it teaches |
|---|---|---|
| mturn-plus5 (multi-turn augmentation) | 5,281 | multi-turn conversation flow |
| eif (exact instruction following) | 5,821 | instruction adherence |
| mentor-ca | 5,047 | mentoring/dialogue |
| dolly-ca | 2,449 | open-ended instruction following |
| coqcat | 2,372 | conversational QA |
| alia-identity | 1,259 | self-identity |
| system-prompt multi-turn | 1,226 | system-prompt adherence |
All conversations CC-BY-4.0 (removes round-1's non-commercial restriction on the training data; note InstruCAT is still in the round-1 lineage). A Catalan system prompt ("Ets un assistent conversacional que respon sempre en català.") was prepended to conversations lacking one. 400 conversations held out for eval.
Training details
- Base:
luispoveda93/MiniCPM5-2B-catalan-chat(round-1 merged model) - Method: LoRA r=32 / α=64, all projection layers, lr 1e-4 cosine (gentler than round 1's 2e-4), warmup 50, 1 epoch, effective batch 32 (4 × 8), max_length 2048 packed, bf16, completion-only loss
- Scale: ~326 packed steps, 2h19m on A100-large
- Observed: train loss 2.44 → ~1.44; eval loss 1.48 → 1.41 → 1.383; token accuracy 0.57 → 0.704 (400-conversation held-out eval)
- Training metrics (trackio)
Artifacts
- This repo: merged bf16 model (5.0 GB)
- Adapter: luispoveda93/MiniCPM5-2B-catalan-chat-v2-lora (on top of the round-1 merged model)
- Round 1: luispoveda93/MiniCPM5-2B-catalan-chat · GGUF
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat-v2", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat-v2")
messages = [
{"role": "system", "content": "Ets un assistent conversacional que respon sempre en català."},
{"role": "user", "content": "Hola! Com estàs avui?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(inputs, max_new_tokens=256)[0][inputs.shape[1]:], skip_special_tokens=True))
Limitations
- 2.5B parameters: limited world knowledge and reasoning depth; Catalan conversational style is the training focus.
- Round 2 used the chat-dense core of the ALIA mixture (~23.4K conversations); the QA corpora (fineweb-edu, RAG, math) were deliberately excluded to prioritize chat behaviour — task-QA skills come from round 1.
- Evaluation is loss/accuracy-based on held-out conversations; no independent Catalan chat benchmark was run.