Dravidian GPT-2 Malayalam

A GPT-2 base model trained from scratch on Malayalam language data, part of the Dravidian GPT-2 series covering Dravidian languages (Malayalam, Telugu, Kannada, Tamil).

Model Details

Training Data

The model was trained on a combination of Malayalam corpora:

Usage

import sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained("pulipakav-1/dravidian-gpt2-malayalam")
model.eval()

# Load tokenizer
model_path = hf_hub_download(
    repo_id="pulipakav-1/dravidian-gpt2-malayalam",
    filename="tokenizer.model"
)
sp = spm.SentencePieceProcessor()
sp.Load(model_path)

# Generate
prompt = "മലയാളം ഭാഷ"
token_ids = sp.Encode(prompt)
inputs = {"input_ids": torch.tensor([token_ids])}

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=0,
        eos_token_id=3,
        bos_token_id=2
    )

print(sp.Decode(outputs[0].tolist()))

Limitations

• The model may generate English text mixed with Malayalam, as web-crawled training data contains code-switched content.
• As a base language model, it is not instruction-tuned and may produce repetitive or incoherent outputs.
• Performance on low-frequency Malayalam vocabulary may be limited.

Citation

If you use this model in your research, please cite:

@misc{dravidian-gpt2-malayalam,
  author = {pulipakav-1},
  title = {Dravidian GPT-2 Malayalam},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/pulipakav-1/dravidian-gpt2-malayalam}
}

License

This model is released under the MIT License.