NovAI Base 102M
NovAI Base 102M is an Indonesian-language causal language model developed by KaryaVirtual.
Created by: Nova Novriansyah, MSc, MBA
Model Details
- Parameters: 102,558,240 (~102.6M)
- Architecture: Llama-style decoder-only Transformer
- Layers: 12
- Hidden size: 720
- Intermediate size: 1,920
- Attention heads: 12
- KV heads: 4
- Vocabulary size: 50,257
- Context length: 1,024 tokens
- Tokenizer: GPT-2 tokenizer
Pretraining
The model was initialized with newly created model weights and pretrained on approximately 2 billion Indonesian-language tokens.
Training hardware:
- NVIDIA A100 40GB
Dataset
The training corpus was prepared from
HuggingFace FineWeb-2,
using the Indonesian Latin (ind_Latn) subset.
Approximately:
- 2 billion training tokens
- 10 million validation tokens
The original dataset is not redistributed with this repository.
Users should refer to the upstream dataset documentation, license, and terms.
Source Repository
Technical documentation and supporting resources:
https://github.com/novrian6/novai
Training
The model was pretrained using causal language modeling.
- Training tokens: ~2 billion
- Validation tokens: ~10 million
- Training steps: 15,259
Full pretraining logs and training source code are available in the project repository: https://github.com/novrian6/novai
Important
This model is not a fine-tuned copy of Llama, GPT, Claude, Gemini, or another pretrained foundation model.
The architecture uses the Llama implementation available through Hugging Face Transformers, while the model weights were newly initialized and pretrained separately for NovAI.
Training Evidence
The documented training run completed at:
15,259 / 15,259 steps
Training loss:
1.7229
Validation loss:
1.7012
Approximate training tokens:
2,000,027,648
Tokenizer
The model uses the GPT-2 tokenizer with a vocabulary size of 50,257 tokens.
Tokenizer: https://huggingface.co/openai-community/gpt2
Intended Use
NovAI Base 102M is intended for:
- Indonesian NLP research
- language-model experimentation
- downstream fine-tuning
- local inference experimentation
- educational and research purposes
Limitations
NovAI Base 102M is a relatively small language model and should not be compared directly with substantially larger commercial or open-weight foundation models.
The model may generate incorrect, repetitive, biased, or factually unreliable content.
This repository contains the base pretrained model and is not an instruction-tuned conversational model.
License
The NovAI Base 102M model weights are released under the Apache License 2.0.
The training dataset, tokenizer, and third-party software remain subject to their respective licenses and terms.
Attribution
Model: NovAI Base 102M
Developer: KaryaVirtual
Author: Nova Novriansyah, MSc, MBA
Website: https://karyavirtual.com/