This is a fine tune of Mistral-7B-v0.1 on a very limited range of Sundanese language datasets that are available. This is a learning project for me where I just wanted to see if it's possible to teach a model a new language that it does not inherently support with just a QLora fine tune. It won't only speak sundanese but it just adds sundanese capability to the model that is to me impressive for the limited data and short amount of training time.
Datasets used: Sundanese sources from this repo that I cleaned, deduped and turned into raw corpus completion in jsonl format.
Training Hyperparameters are determined from trial and error multiple times and inspired by the Chinese LLaMa-Alpaca finetune
-
https://github.com/ymcui/Chinese-LLaMA-Alpaca/wiki/Training-Details#preparation-vocabulary-expansion
-
load_in_4bit: true
-
adapter: qlora
-
sequence_len: 1024
-
lora_r: 8
-
lora_alpha: 32
-
lora_dropout: 0.05
-
lora_target_modules: gate_proj, down_proj, up_proj, q_proj, v_proj, k_proj, o_proj
-
gradient_accumulation_steps: 2
-
micro_batch_size: 16
-
num_epochs: 2
-
optimizer: adamw_bnb_8bit
-
lr_scheduler: cosine
-
learning_rate: 0.0002
Hardware/Software used:
- Windows WSL2
- Axolotl
- RTX 4090 24GB
Parameters for example chat:
- temperature: 0.5
- top_p: 0.9
- top_k: 20
- repetition_penalty: 1.05
Character for example chat:
- Context: Di handap ieu obrolan dina basa Sunda antara User jeung AI. Ieu te percakapan bahasa sunda jeung AI Large Language Model nu nga bantu jelema. AI ngan ngabales ku basa sunda. AI na jago ngajawab pertanyaan nu hese jeung mere saran. AI nuturkeun kahayang User.
- Greeting: Naon anu abdi tiasa ngabantosan anjeun?
Example chat:
-
AI: Naon anu abdi tiasa ngabantosan anjeun?
-
You: pang jelaskeun naon balapan mobil formula 1 te
-
AI: Formula 1 mangrupa salah sahiji kompetisi mobil anu diadegkeun taun 1946, anu dipingpin ku Fédération Internationale de l'Automobile (FIA). Kompetisi ieu dilaksanakeun unggal usum ti bulan Maret nepi ka Nopémber. Balapan Formula 1 biasana dilakukeun di laporan anu panjangna kurang leuwih 5-7 km, anu mana mobil-mobil anu parigel dibawa ku tim-tim anu aya di dunia.
-
You: Ai fernando alonso te saha? Pang terangkeun
-
AI: Fernando Alonso nyaéta saorang pembalap asal Spanyol anu lahir tanggal 29 Juli 1981 di Oviedo, Asturias, Spanyol. Anjeuna mimiti karir profesionalna di taun 1999, nalika anjeuna meunang gelar juara GP2 Series taun 2000. Dina taun 2001, Alonso mimiti maén di Formula One, nalika anjeuna ditandatanganan kontrak ku Tim Minardi. Tapi, sabada dua kali balapan, anjeuna diganti ku Alexander Wurz. Taun 2003, Alonso ganti tim ka Tim Renault, anu mana anjeuna bisa nyitak rékor anyar minangka pembalap paling umurna anu bisa meunang podium di Grand Prix Monaco. Podium éta ogé jadi hiji-hijina podium anu didamel ku Alonso di musim 2003.