Cambodia Tourism LM (276M Scratch)
Model Description
CambodiaTourismLM is a custom bilingual language model built completely from scratch, designed to understand and generate text related to Cambodian tourism, culture, geography, and history in both English and Khmer.
Instead of fine-tuning an existing foundation model, this model was trained from random initialization using a custom-built Byte-Level BPE tokenizer and a Llama-style causal language model architecture.
- Developed by: Chhorn Seyha
- Model type: Causal Language Model (Transformer Decoder)
- Language(s) (NLP): English, Khmer (
en,km) - License: MIT
- Architecture: Llama-based (Custom 276M parameter configuration)
Model Architecture
The model uses a scaled-down Llama configuration optimized for custom local training:
- Total Parameters: ~276,000,000
- Hidden Size: 1024
- Intermediate Size: 4096
- Number of Layers: 16
- Number of Attention Heads: 16
- Max Position Embeddings: 512
Training Details
- Tokenizer: Custom Byte-Level BPE Tokenizer trained on a bilingual corpus with a vocabulary size of 32,000.
- Dataset Size: 10,000 bilingual articles (English and Khmer).
- Total Tokens Trained: 39,000,000 tokens.
- Hardware: Trained locally on an NVIDIA RTX 4070 GPU using PyTorch.
- Final Training Loss: 3.533
Usage
You can load this model directly using the Hugging Face transformers library:
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
from tokenizers import ByteLevelBPETokenizer
# Load tokenizer
tokenizer_path = "seyhachhorn/cambodia-tourism-lm" # Update with your repo id if needed
model = LlamaForCausalLM.from_pretrained("seyhachhorn/cambodia-tourism-lm")
input_text = "Angkor Wat is"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))