LiLM1-230M-base
LiLM1-230M-base is the pretrained LiLM1 checkpoint. It has 233,897,728 parameters and a 4,096-token context. The public name uses the nearest 10M parameter class.
This is a base model. It is not instruction-tuned.
Model details
| Field | Value |
|---|---|
| Developer | Akshay Gautam |
| Model repository | glouriousgautam/LiLM1-230M-base |
| Parameters | 233,897,728 |
| Context length | 4,096 tokens |
| Architecture | Hybrid decoder: 10 gated causal-convolution blocks and 6 GQA blocks |
| Pretraining | 40B token exposures |
| Standardized training compute | 75.1 to 75.6 GPU-hours on one RTX PRO 6000 Blackwell |
| Tokenizer | SmolLM2 135M tokenizer, revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2 |
| Source checkpoint SHA-256 | a0c7627df310a21ac1ceeedbce3694084a6c8e36deb35bbd2252fb682550d963 |
| Safetensors SHA-256 | 2964530e470d8e1b942e5c9ad8d67bd3387a07f358831e1fdf785adfef151908 |
| License | Apache-2.0 |
The hybrid stack uses width 1,024, SwiGLU intermediate size 2,560, and tied input and output embeddings. Convolution layers use kernel size 3. Attention layers sit at depths 2, 5, 8, 11, 13, and 15. Attention uses 16 query heads, 4 key/value heads, head dimension 64, and Q/K RMSNorm. Position encoding is RoPE with base theta 10,000.
Training
The base model used 40B token exposures at 4,096 context. Training used BF16 on one NVIDIA RTX PRO 6000 Blackwell GPU. The recorded steady rate was 147K to 148K tokens per second. This gives a standardized pretraining estimate of 75.1 to 75.6 GPU-hours.
The public pretraining corpus is not available.
Evaluation
The 40B base checkpoint scores 45.22% on an 11-task zero-shot aggregate. WikiText word perplexity is 26.71. WikiText bits per byte is 0.886.
Same-scale public references remain ahead on this aggregate. SmolLM2 135M scores 47.84%. LFM2.5 230M scores 48.73%. SmolLM2 360M scores 54.02%. LiLM1 used 50 times fewer pretraining tokens than SmolLM2 135M.
The base checkpoint is a next-token model. It is not a tool-use or instruction-following model.
Load
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "glouriousgautam/LiLM1-230M-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo,
trust_remote_code=True,
dtype=torch.bfloat16,
)
The architecture is custom. Loading requires trust_remote_code=True.
Intended use
Use LiLM1-230M-base for research on compact language models. It can serve as the parent for further training. Validate any downstream use before deployment.
Do not use this checkpoint as a chat or tool-calling model. Do not use it with context longer than 4,096 tokens. Do not use it for medical, legal, financial, or security decisions.
Limitations
- English is the only evaluated language.
- The model has limited math, retrieval, and instruction-following ability.
- The evaluation does not cover every downstream task.
- No quantized release has passed final validation.
License
Apache License 2.0.
Citation
@misc{lilm1_base_2026,
title = {LiLM1-230M-base},
author = {Akshay Gautam},
year = {2026},
howpublished = {https://huggingface.co/glouriousgautam/LiLM1-230M-base}
}