CALI
CALI (Computer Assistant Lightweight Intelligence) is an experimental lightweight language model trained from scratch on a limited-scale bilingual dataset consisting of Indonesian and English text.
The model was developed to explore lightweight transformer architectures, small-model efficiency, and language model training under constrained computational resources and limited datasets.
CALI is NOT a large-scale foundation model and was NOT trained on massive internet-scale datasets like modern commercial language models.
Important Notes
Due to the relatively small pretraining corpus, the model may exhibit noticeable bias toward the most recent or dominant domains seen during training. Continued pretraining, fine-tuning, or alignment is strongly recommended depending on the intended use case.
Model Details
| Property | Value |
|---|---|
| Parameters | 121M |
| Layers | 11 |
| Hidden Size | 768 |
| Attention Heads | 4 |
| KV Heads | 1 |
| Head Dimension | 192 |
| FFN Dimension | 2304 |
| Context Length | 1024 |
| Vocabulary Size | 32000 |
Architecture
For a detailed explanation of the CALI architecture:
Pretraining
CALI was trained entirely from scratch using carefully selected and filtered datasets designed for research purposes rather than maximizing dataset size.
The training corpus includes:
- English text
- Indonesian text
- Wikipedia
- News articles
- General documents
- Source code
Benchmark Results
Below is the performance comparison of CALI-0.1B against several prominent Small Language Models (SLMs) in the 100M+ parameter range.
| Model Name | Piqa | MMLU Math | ARC-Challenge | HellaSwag |
|---|---|---|---|---|
| CALI-0.1B | 54.19% | 28.04% | 24.66% | 27.00% |
| SmolLM2-135M | 58.50% | 29.90% | 31.10% | 43.20% |
| GPT-X2-125M | 51.60% | 27.80% | 27.80% | 40.50% |
| SmolLM-135M | 56.30% | 28.80% | 28.80% | 42.70% |
| MobileLLM-R1-140M-base | 49.90% | 24.70% | 24.70% | 33.90% |
| GPT-X-125M | 50.80% | 26.70% | 26.70% | 36.50% |
| GPT-2 (124M) | 39.50% | 22.60% | 22.60% | 31.50% |
| GPT-Neo-125M | 39.40% | 22.90% | 22.90% | 30.40% |
| OPT-125M | 40.20% | 22.90% | 22.90% | 31.40% |
Note: The CALI-0.1B scores are strict raw accuracies (acc / acc_norm) obtained directly from the evaluation tracker logs.
Training Progress
| Tokens | Step | Final Loss |
|---|---|---|
| 250M | 13,564 | 3.53 |
| 350M | 18,989 | 3.53 |
| 450M | 24,415 | 4.69 |
| 614M | 33,356 | 2.71 |
Notes
- Experimental transformer architecture
- Requires a custom inference implementation
- Uses Grouped-Query Attention (GQA)
- Intended for research and experimentation
Citation
If you use or reference this model in your research or projects, please cite:
@article{cali2026,
title = {CALI 0.1B},
author = {Sandroeth},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/Sandroeth/cali-0.1B}
}
Author
Sandroeth