Fina-30M v0.1 (base)
Fina-30M is a 29,368,832-parameter (~29.4M) decoder-only causal Transformer for Indonesian, trained from scratch end-to-end on Google Colab free tier. This repository hosts the base pretrained checkpoint (not instruction-tuned) — it continues text, it does not follow chat-style instructions.
⚠️ License note: training data mixes Wikipedia ID, a general web corpus (Corpus-Indonesia), and an Indonesian news dataset that carries a CC BY-NC 4.0 (non-commercial) restriction. This repo is tagged
cc-by-nc-4.0accordingly. The licenses of the other two source datasets were not independently verified by the author — check them yourself before any commercial use.
Model summary
| Parameters | 29,368,832 (~29.4M), embedding tied with LM head |
| Architecture | Decoder-only causal Transformer, 8 layers, d_model 512, 8 heads (head_dim 64), SwiGLU FFN (d_ff 1024), RoPE, RMSNorm, no biases |
| Context length | 512 tokens |
| Vocabulary | 16,384 (byte-level BPE, 5 special tokens) |
| Training objective | Causal language modeling (next-token prediction) |
| Status | Base pretraining complete; standard benchmarks (FINA-Bench) not yet run |
Training data
Cleaned and deduplicated corpus (98/2 train/val split by content hash), 624.0M training tokens:
| Source | Domain | Share of train tokens | Notes |
|---|---|---|---|
Wikipedia ID (wikimedia/wikipedia, 20231101.id) | Encyclopedic | 40.0% (249.7M) | 572,737 / 665,622 docs retained |
Corpus-Indonesia (Lyon28/Corpus-Indonesia) | Web / forums | 57.8% (360.8M) | Noisy; 6.25M / 19.5M lines retained after cleaning |
Indonesian News (iqballx/indonesian_news_datasets) | News | 2.2% (13.5M) | Tempo, CNN Indonesia, CNBC Indonesia, JawaPos, Kumparan, Okezone, Suara; ~CC BY-NC 4.0 |
Cleaning: custom stopword-based Indonesian language filter (fastText was unavailable in the training
environment), boilerplate/HTML/code stripping, exact + near-duplicate removal, source-specific artifact
stripping. IndoMMLU was downloaded but used only for evaluation planning, never mixed into training.
Sampled during training at a fixed mixture of wiki 40% / web 57% / news 3% (≈ natural token distribution), for 2 epochs / 38,087 steps (≈1.25B tokens seen, ≈21.2 tokens/parameter/epoch — near Chinchilla-optimal for one epoch).
Training procedure
- Optimizer: AdamW, peak LR 6e-4 → 6e-5 (cosine schedule, 500-step warmup), weight decay 0.1, grad clip 1.0
- Batch: 64 × 512 = 32,768 tokens/step
- Precision: bf16 or fp16 autocast (varied by assigned Colab GPU)
- Checkpoint/resume to Google Drive every 500–1,000 steps (training survived multiple free-tier disconnects and GPU quota resets across ~2 days wall-clock)
Evaluation
Validation loss / perplexity by domain (fixed mixture weights 40/57/3):
| Domain | Loss | Perplexity |
|---|---|---|
| Wikipedia | 2.94 | 19.0 |
| News | 3.06 | 21.4 |
| Web | 3.73 | 41.8 |
| Mixed | 3.40 | 29.9 |
Final training loss: 3.25. No standard benchmark (MMLU/IndoMMLU, HellaSwag, ARC, etc.) has been run yet.
Intended use & limitations
This is a small-scale research/educational model, not intended for production use.
- Base model only — no instruction tuning, no RLHF, no safety fine-tuning. It continues text; it does not reliably follow instructions or hold a conversation.
- Weak factual recall. With only ~29M parameters, the model hallucinates facts confidently (e.g. it has been observed to name the wrong Indonesian president) and can self-contradict. This is treated as a capacity limitation, not a training bug — do not use this model as a source of factual information. Consider pairing it with retrieval (RAG) if factual grounding matters.
- Mild repetition on rare or technical topics.
- Context length is capped at 512 tokens.
- No decontamination check was performed against evaluation benchmarks.
- Trained with a single seed per configuration; no variance estimates are available.
How to use
This model uses a custom architecture (not a standard transformers model class), so loading requires
the model code from this repo (modeling_fina.py) rather than AutoModel.
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import torch, json
repo_id = "Dawgggggg/fina-30m-v0.1"
cfg = json.load(open(hf_hub_download(repo_id, "config.json")))
weights_path = hf_hub_download(repo_id, "model.safetensors")
modeling_path = hf_hub_download(repo_id, "modeling_fina.py")
# import Fina + Cfg from the downloaded modeling_fina.py, then:
# model = Fina(Cfg(**cfg))
# sd = load_file(weights_path)
# model.load_state_dict(sd, strict=False) # strict=False: head.weight is tied to tok.weight, not stored separately
# model.eval()
Tokenizer: load tokenizer.json (byte-level BPE, vocab 16,384) with the Hugging Face tokenizers library.
Citation
If you use this model, please cite:
@techreport{fina30m2026,
title = {Fina-30M: Design, Training, and Experimental Analysis of a Small Indonesian Language Model},
author = {Fadhlan (Horii)},
year = {2026},
note = {Technical research paper, v0.1},
url = {https://github.com/muhfadtz}
}
Acknowledgements
Built with the Hugging Face tokenizers and datasets libraries, and PyTorch. Training data drawn from
wikimedia/wikipedia, Lyon28/Corpus-Indonesia, and iqballx/indonesian_news_datasets on the Hugging
Face Hub. Entirely trained on Google Colab free tier + Google Drive storage.