Dravidian GPT-2 Kannada
A GPT-2 base model trained from scratch on Kannada language data, part of the Dravidian GPT-2 series covering Dravidian languages (Malayalam, Telugu, Kannada, Tamil).
Model Details
| Property | Value |
|---|---|
| Architecture | GPT-2 (Causal LM) |
| Parameters | 110M |
| Layers | 12 |
| Attention Heads | 12 |
| Hidden Size | 768 |
| Context Length | 1024 |
| Vocab Size | 32,000 |
| Tokenizer | SentencePiece BPE |
Training Data
The model was trained on a combination of Kannada corpora:
| Source | Description |
|---|---|
| CC-100 | Web-crawled Kannada text from CommonCrawl |
| Wikipedia (20231101.kn) | Kannada Wikipedia dump (November 2023) |
| Samanantar | Kannada parallel corpus |
| TinyStories (kn) | Cleaned Kannada short stories |
Sample Outputs
| Prompt | Generated Text |
|---|---|
| ಕರ್ನಾಟಕದ | ಕರ್ನಾಟಕದ ಪ್ರವಾಹ ಸಂತ್ರಸ್ತರಿಗೆ ನೆರವಾಗುವುದು ಹೇಗೆ? ಇಲ್ಲಿದೆ ಮಾಹಿತಿ |
| ಕನ್ನಡ ಸಾಹಿತ್ಯ | ಕನ್ನಡ ಸಾಹಿತ್ಯ ಚರಿತ್ರೆ ಒಂದು ಅಧ್ಯಯನ, ಇತಿಹಾಸ ಮತ್ತು ಸಂಸ್ಕೃತಿ ಇಲಾಖೆ ಕರ್ನಾಟಕ ಸರ್ಕಾರ |
| ಭಾರತದ | ಭಾರತದ ಟಾಪ್ ಅತಿ ಶ್ರೀಮಂತ ರಾಜ್ಯಗಳು ಯಾವುವು ಗೊತ್ತೆ? |
Usage
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
# Load model
model = AutoModelForCausalLM.from_pretrained("pulipakav-1/dravidian-gpt2-kannada")
model.eval()
# Load tokenizer
model_path = hf_hub_download(
repo_id="pulipakav-1/dravidian-gpt2-kannada",
filename="tokenizer.model"
)
sp = spm.SentencePieceProcessor()
sp.Load(model_path)
# Generate
prompt = "ಕನ್ನಡ ಭಾಷೆ"
token_ids = sp.Encode(prompt)
inputs = {"input_ids": torch.tensor([token_ids])}
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=50,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.2,
pad_token_id=0,
eos_token_id=3,
bos_token_id=2
)
print(sp.Decode(outputs[0].tolist()))
Limitations
- The model may generate English text mixed with Kannada, as web-crawled training data contains code-switched content.
- As a base language model, it is not instruction-tuned and may produce repetitive or incoherent outputs.
- Performance on low-frequency Kannada vocabulary may be limited.
Citation
If you use this model in your research, please cite:
@misc{dravidian-gpt2-kannada,
author = {pulipakav-1},
title = {Dravidian GPT-2 Kannada},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/pulipakav-1/dravidian-gpt2-kannada}
}
License
This model is released under the MIT License.