🇱🇰 SinhalaLM — Instruct

A fully fine-tuned, instruction-following Sinhala question-answering model

SinhalaLM-Instruct · by HelaAI

📑 Table of contents

• Overview
• Relationship to the base model
• Prompt template
• Architecture
• Fine-tuning
• How to use
• Intended use
• Limitations and biases
• Files in this repo
• License
• Citation
• Contact

🔎 Overview

SinhalaLM — Instruct is an instruction-tuned, question-answering variant of SinhalaLM, a GPT-style decoder-only transformer built from scratch for the Sinhala language. It combines the same core architectural ingredients as the base model — Rotary Position Embeddings (RoPE), SwiGLU gated feed-forward blocks, and a custom Sinhala byte-pair encoding (BPE) tokenizer — with full fine-tuning on a Sinhala question-answer dataset, so that every parameter in the model (not just a low-rank adapter subset) adapts to the instruction-following QA task.

Given a Sinhala question wrapped in a fixed instruction template, the model generates a Sinhala answer directly, without requiring any few-shot examples at inference time.

This is a custom architecture — it is not a standard transformers (AutoModel) class. The full model definition, tokenizer, and prompt-building utilities ship in this repository as modeling_sinhalalm.py and must be imported directly (see How to use below).

⚠️ Private repository. Only Hugging Face accounts explicitly granted access can pull this repo. Do not share the access token.

🔗 Relationship to the base model

SinhalaLM — Instruct is obtained by fully fine-tuning the private base checkpoint ChamaraVishwajithRajapaksha/SinhalaLM-V.0.0.1 — a GPT-style Sinhala language model pretrained from scratch — on Sinhala question-answer data. "Full" fine-tuning means every weight in the network is updated during this stage, rather than freezing the backbone and training a small set of adapter parameters (e.g. LoRA). This is practical here specifically because the base model is small (tens of millions, not billions, of parameters), which keeps full fine-tuning affordable on a single GPU while still letting every layer specialize to the QA task.

📝 Prompt template (Alpaca-style, Sinhala)

The model was fine-tuned on prompts built like this — use the exact same format at inference time, and stop generating at <EOS>:

පහත ප්‍රශ්නයට සිංහලෙන් පිළිතුරු දෙන්න.

### ප්‍රශ්නය:
<your question here>

### පිළිතුර:
<model completes from here>

Loss was masked during training so that only the answer tokens (after ### පිළිතුර:) contributed to the loss — the model was never trained to reproduce the question or the system instruction header, only to continue them with an appropriate answer.

🏗️ Architecture

SinhalaLM — Instruct inherits its decoder-only transformer architecture directly from the SinhalaLM base model.

Note: the instruct checkpoint's parameter count (~44.7M) is larger than the base model's (~32.9M); the additional parameters come from task-specific components added during fine-tuning on top of the shared transformer backbone.

🎯 Fine-tuning

Full fine-tuning curves and a CSV/JSON breakdown are provided in analytics/. Example generations from the fine-tuning run are in sample_generations.txt (if included in this repo).

🚀 How to use

Install dependencies:

pip install torch huggingface_hub safetensors

Download, load, and ask a question:

from huggingface_hub import snapshot_download
import torch, json, sys

# 1. Download the repo (private — requires a token with read access)
local_dir = snapshot_download(repo_id="ChamaraVishwajithRajapaksha/SinhalaLM-V.0.0.1-it", token="YOUR_HF_TOKEN")

# 2. Import the model + tokenizer classes shipped in this repo
sys.path.insert(0, local_dir)
from modeling_sinhalalm import SinhalaGPT, FastBPETokenizer, build_prompt, get_prompt_prefix

# 3. Load config
config = json.load(open(f"{local_dir}/config.json"))

# 4. Load tokenizer
tokenizer = FastBPETokenizer.load(f"{local_dir}/bpe_vocabulary.json")

# 5. Build model + load fine-tuned weights
from safetensors.torch import load_file
state_dict = load_file(f"{local_dir}/model.safetensors")

model = SinhalaGPT(
    vocab_size=config["vocab_size"],
    embed_dim=config["n_embed"],
    n_heads=config["n_heads"],
    n_layers=config["n_layers"],
    block_size=config["block_size"],
    dropout=0.0,  # no dropout at inference
)
model.load_state_dict(state_dict)
model.eval()

# 6. Ask a question using the exact fine-tuning prompt template
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

question = "ශ්‍රී ලංකාවේ අගනුවර කුමක්ද?"
prefix = get_prompt_prefix(question)
ids = [tokenizer.stoi.get("<BOS>", 2)] + tokenizer.encode(prefix)
ctx = torch.tensor([ids], dtype=torch.long, device=device)

out = model.generate(ctx, max_new_tokens=200, temperature=0.7, top_k=50)
answer_ids = out[0, len(ids):].tolist()
print(tokenizer.decode(answer_ids))

Generation parameters

⚠️ Always build the prompt with get_prompt_prefix() (or the exact template shown above) — deviating from the fine-tuning prompt format will noticeably degrade answer quality, since the model was only trained to continue that specific structure.

🎯 Intended use

SinhalaLM — Instruct is intended for Sinhala question answering in research and prototyping contexts — e.g. exploring compact, from-scratch, instruction-tuned models for a low-resource language, or as a lightweight QA component in a larger Sinhala NLP pipeline.

Not intended for:

• ❌ Production deployment without further evaluation, safety review, and human oversight.
• ❌ High-stakes factual use (medical, legal, financial, or safety-critical questions) — answers are not verified or fact-checked.
• ❌ Multi-turn dialogue or open-ended chat — the model is fine-tuned specifically for single-turn question → answer completion in the prompt format above.
• ❌ Any use requiring inputs/outputs longer than 512 tokens total.

⚠️ Limitations and biases

• Fine-tuned on a relatively small QA dataset (447 training steps) — coverage of topics outside the fine-tuning distribution may be shallow or inaccurate.
• As with any fine-tuned language model, answers may be fluent but factually incorrect ("hallucinated") — always verify important claims independently.
• Inherits any biases present in both the base pretraining corpus and the QA fine-tuning data.
• Context window is limited to 512 tokens (prompt + answer combined).
• Strict reliance on the exact prompt template — inputs that deviate from the trained format may produce degraded or off-format answers.
• No safety filtering or content moderation has been applied to outputs.
• Monolingual (Sinhala) by design — not intended for multilingual or code-switched input.

📁 Files in this repo

📜 License

This model is released under a custom/other license. Update this section with your actual terms (e.g. permitted uses, redistribution, commercial use) before granting access to others.

📚 Citation

If you use this model, please cite:

@misc{sinhalalminstruct,
  title  = {SinhalaLM -- Instruct},
  author = {Chamara Vishwajith and Dr. Kushan Sudheera and Dineth Jayakody},
  year   = {2026},
  note   = {Instruction-tuned, fully fine-tuned Sinhala question-answering model, built on the SinhalaLM base checkpoint}
}

📬 Contact

Developed by HelaAI. For questions or access requests, contact one of the authors of the base SinhalaLM model, or the repository owner on Hugging Face.

—— SinhalaLM · Instruct · HelaAI ——