sinhala-llama-1b-sft-diverse

Model Description

An instruction-tuned Sinhala language model fine-tuned on Minuri/sinhala-llama-1b-corpus-diverse (diversity-optimised continually pretrained LLaMA 3.2 1B). This is the best-performing model in the series, built on top of the CPT model that achieved the lowest perplexity (10.49). Part of a diversity-driven Sinhala language model adaptation study.

SFT model variants in this series:

• Minuri/sinhala-llama-1b-sft-baseline - SFT on base LLaMA 3.2 1B (no CPT)
• Minuri/sinhala-llama-1b-sft-news - SFT on sinhala-llama-1b-corpus-news
• Minuri/sinhala-llama-1b-sft-random - SFT on sinhala-llama-1b-corpus-random
• Minuri/sinhala-llama-1b-sft-diverse - SFT on sinhala-llama-1b-corpus-diverse - this repo

This model is the result of supervised fine-tuning (SFT) of Minuri/sinhala-llama-1b-corpus-diverse on the Minuri/sinhala-sft-dataset (~213K Sinhala instruction pairs). The sinhala-llama-1b-corpus-diverse model was continually pretrained on a diversity-optimised Sinhala corpus prior to SFT, achieving a perplexity of 10.49 - the lowest among the three CPT variants.

Training Data

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Minuri/sinhala-llama-1b-sft-diverse")
model = AutoModelForCausalLM.from_pretrained("Minuri/sinhala-llama-1b-sft-diverse")

Intended Uses

• Sinhala instruction following
• Low-resource NLP research and benchmarking
• Base for further Sinhala LLM research

Limitations

• 1B parameter model with limited reasoning capability

Related Repositories

License

This model is derived from meta-llama/Llama-3.2-1B and is subject to the LLaMA 3.2 Community License.