KW5-Lite Instruct

A 109.5M-parameter Swahili instruction-following model, LoRA fine-tuned from regnant-io/kw5-lite-base.

Built by Regnant. Training code: kw5-lite and another 88K examples.

If you used a previous revision of this repo, please re-pull. The chat template shipped before this revision inserted a second <s>[INST] when a system message was present and appended a trailing space at generation time. On the same weights that cost roughly half the model's quality — see What changed.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-lite-instruct")
tok = AutoTokenizer.from_pretrained("regnant-io/kw5-lite-instruct")

messages = [{"role": "user", "content": "Orodhesha majina ya miji mitano ya Tanzania."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# add_special_tokens=False: the template already emits <s>.
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**inputs, max_new_tokens=256)   # generation_config carries good defaults
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

1. Dar es Salaam
2. Dodoma
3. Arusha
4. Tanga
5. Zanzibar

Decoding settings matter a lot at this size

This is a 110M model. It is coherent at low temperature with a repetition penalty and degenerates at the transformers defaults. The shipped generation_config.json already sets:

temperature 0.1–0.2 with repetition_penalty 1.3–1.4 is the usable band. Above ~0.7, or with no repetition penalty, output quality falls off sharply.

Prompt format

<s>[INST] {user} [/INST] {assistant}</s>[INST] {user2} [/INST] {assistant2}</s>

with an optional system block inside the first [INST] only:

<s>[INST] <<SYS>>
{system}
<</SYS>>

{user} [/INST]

apply_chat_template produces exactly this. Two rules if you build prompts by hand:

1. Tokenize with add_special_tokens=False. The template emits <s> itself; letting the tokenizer add another gives a doubled BOS the model never saw. add_bos_token is pinned to false in tokenizer_config.json for this reason.

2. Do not use <|system|> / <|user|> / <|assistant|> (token ids 4–6). They were reserved when the tokenizer was trained but never appear in the pretraining corpus, and LoRA left embed_tokens frozen — so those embedding rows are still at initialisation. Prompting with them returns noise:

   <|user|>\nOrodhesha majina ya miji...   ->  'KANIWA AMUAAAAAAAAAAAAAHAAAAAAAAATI...'
   <s>[INST] Orodhesha majina ya miji... [/INST]  ->  '1. Dar es Salaam\n2. Dodoma\n...'
   

Model details

Fine-tuning

The rank and adapted-module list above are recovered directly from the published weights: the singular-value spectrum of (merged − base) cuts off sharply after 16 for every adapted projection, and the non-adapted tensors match the base checkpoint bit-for-bit.

Training data composition

~107,000 examples spanning instruction-following, multi-turn dialogue, Tanzanian factual and cultural content (methali, mila, utamaduni), Swahili grammar (sarufi), Sheng code-switching, safety and refusal patterns, and identity responses. Loss was masked to assistant turns only — prompts do not contribute to the objective.

Evaluation

Generation-based instruction-following, 16 prompts across five categories, graded by keyword and structure checks (scripts/eval_sft.py in the training repo), at temperature 0.2 / repetition penalty 1.3:

This is a small, non-standard suite; treat it as a smoke test, not a benchmark.

Read the failure column. Factual recall is the clear weak point — the model does not reliably know that the capital of Tanzania is Dodoma, or that Lake Victoria is Africa's largest lake. At 110M parameters trained on 1.41B tokens, that is expected. Use this model for fluent Swahili generation, formatting and conversational structure; do not use it as a knowledge source.

What changed in this revision

Two things, both of which affected everyone using this model:

1. The chat template was wrong. Measured on identical weights and prompts:

The old template emitted <s>[INST] before every user turn rather than only the first, so a system message produced <s>[INST] <<SYS>>…<</SYS>>\n\n<s>[INST] {user} [/INST] — a second BOS in the middle of the prompt — and multi-turn produced </s><s>[INST] where training used </s>[INST]. add_generation_prompt also appended a bare space, which changes how the first generated word tokenizes.

2. The weights are now the model that actually matches this repo's stated base. The previous revision was fine-tuned from an unpublished base checkpoint (step 8,176) that is past the point where pretraining began overfitting, on ~22,500 examples. This revision is fine-tuned from the published kw5-lite-base (step 6,150, exactly 2 epochs) on ~107,000 examples. Non-LoRA tensors now match kw5-lite-base exactly, so the two repos are finally consistent.

Also in this revision: embeddings re-tied (536 MB → 219 MB, numerically identical — lm_head was a byte-for-byte duplicate of embed_tokens), pad_token_id corrected to 3, and useful generation defaults shipped.

Limitations

• Factual reliability is poor. See the evaluation section. Do not use for factual lookup, medical, legal or financial advice.
• Count adherence is weak. Asked for five items it will sometimes produce ten.
• Primarily Tanzanian Swahili. Coverage of Kenyan, Ugandan and Congolese varieties is thinner.
• 2048-token context, no tool use, no code, English capability is incidental and untested.
• Safety tuning is minimal — a few hundred refusal examples. Not suitable for unsupervised deployment to end users.
• Trained on web-scraped text (FineWeb-2) and so carries its biases.

Citation

@misc{kw5lite2026,
  title  = {KW5-Lite: a 110M-parameter Swahili language model trained on a single T4},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-lite-instruct}
}

Apache 2.0.