Tamillama_Tiny: A 30M tiny llama model trained to tell stories in Tamil
TL;DR:
This is an experimental model inspired by the paper https://arxiv.org/abs/2305.07759 - How Small Can Language Models Be and Still Speak Coherent English?.
Extended the same concept for Tamil. A 30M parameter LLaMA architecture model that outputs coherent Tamil is preseted here.
Additional experimentation which is included in the model:
- This is a multilanguage model as it can output both English and Tamil stories.
- The model also does translation of stories from Engish to tamil and vice versa. To see the translation feature, set the max_new_tokens > 512.
- Translation of original stories from the tinystories dataset was done using IndicTrans
For now, this is a toy model for researchers, students and LLM enthusiasts to play with the linquistic capability of the model.
Weights Release, License and Usage
We release the weights in two formats: Hugging Face transformers format and GGML format to use with CTransformers or LLaMA.cpp.
This is not fit for any practical purpose other than for research/experimentation use cases.
Usage:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("RajuKandasamy/tamillama_tiny_30m")
model = AutoModelForCausalLM.from_pretrained("RajuKandasamy/tamillama_tiny_30m")
prompt = f"""சொற்கள்:
வாக்குறுதி, எலி, பெரியது
சுருக்கம்:"""
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
generation_output = model.generate(
input_ids=input_ids, max_new_tokens=256
)
print(tokenizer.decode(generation_output[0]))