lilo-100m — Lightweight Local Intelligence & Optimizer
A ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following in Indonesian. This is the lilo release of neosantara/wader-100m: same architecture and weights, re-packaged explicitly for CPU-only, low-memory, low-footprint use.
- Base: neosantara/wader-100m-base · pretrained on neosantara/IndoTalk
- Upstream SFT: neosantara/wader-100m on neosantara/IndoTalkSFT
- Trained from scratch — no weights from DeepSeek-V4 were used.
Why lilo
| wader-100m (original) | lilo-100m | |
|---|---|---|
| Weights on disk | 421MB fp32 | 119MB (int8) / 221MB (fp16) |
| CPU-friendly runtime | bf16 NaN bugs, no KV cache | fp32/fp16 stable, KV cache, inference_mode |
| Runs on 2 vCPU / 2GB RAM VPS | untested | measured |
Variants
| Variant | File | Disk | PPL delta vs fp32 | Greedy outputs |
|---|---|---|---|---|
fp16 (default) | model.fp16.safetensors | 221MB | +0.08% | 5/5 identical |
int8 | model.int8.safetensors | 119MB | +0.32% | 5/5 identical |
Notes:
int8is symmetric group-wise quantization (group=64) on allnn.Linearlayers + embeddings; norms/router/HC params stay fp32. Scales cost ~7MB.- Both dequantize to fp32 compute on load — the disk footprint is what shrinks, because this CPU (Zen2, AVX2-only, no VNNI) does fp32 GEMM as fast as int8 anyway, and fp16 GEMM is faster still.
- fp16 is the default because it is both the closest to the original quality and the fastest on CPU.
CPU speed (measured on 2 vCPU AMD EPYC 7K62 "Zen2", AVX2-only, box under load)
| Variant | threads=1 | threads=2 |
|---|---|---|
| fp16 | 14–22 tok/s | 11–12 tok/s |
| int8 | 9–19 tok/s | 5–8 tok/s |
On this tiny model (320 hidden, 8 layers) thread sync overhead dominates — --threads 1 is fastest. On 8+ core machines try 2–4.
Memory safety (no OOM)
Peak RSS measured on a 2 vCPU / 1.9GB RAM VPS: ~0.85GB (int8) / ~1.05GB (fp16) including the Python/torch runtime. Rules baked into run_cpu.sh:
OMP_NUM_THREADSbounded,MALLOC_ARENA_MAX=1(glibc per-thread arenas can silently eat GBs)TMPDIRforced to disk, never/tmpwhen/tmpis tmpfs- prefill chunked at 256 tokens so attention never allocates a big
[S,S]matrix
Usage
git lfs install
git clone https://huggingface.co/hzlo0/lilo-100m
cd lilo-100m
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers safetensors tokenizers accelerate
bash run_cpu.sh # chat, default fp16
CUDA_VISIBLE_DEVICES="" python chat_lilo.py --variant int8 --threads 1 "Halo, apa kabar?"
Chat template (DeepSeek-V4 style):
<|begin_of_sentence|><|User|>Halo<|Assistant|>
import torch
from lilo_common import load_lilo, generate_lilo, chat
torch.set_num_threads(1)
tok, model = load_lilo(".", variant="fp16") # or "int8"
print(chat(tok, model, [{"role": "user", "content": "Halo, apa kabar?"}],
max_new_tokens=200, temperature=0.7, top_p=0.9))
Requires trust_remote_code-free loading: the model code ships as plain files (modeling_deepseek_v4.py) loaded by lilo_common.py. (The same code also works via AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True).)
Architecture
| Component | Details |
|---|---|
| Parameters | ~110M (41M embeddings, 69M non-embedding) |
| Hidden size | 320 · Layers 8 |
| Attention | 8 heads, 1 KV head (MQA), MLA q_lora_rank=160, grouped low-rank output |
| MoE | 4 routed experts + 1 shared, top-2 routing |
| Hyper-Connections | hc_mult=4, Sinkhorn routing (replaces residual stream) |
| Vocab | 129,280 (DeepSeek-V4 tokenizer) · Context 2,048 |
What was fixed for CPU in this release
- freqs_cis recomputed on demand — original code left a buffer uninitialized after meta-device loading, corrupting RoPE beyond a few tokens.
- KV cache honored — original
forwardhardcodeduse_cache=False, recomputing the whole sequence every decode step. This release caches and steps one token. return_dictdefaults to True — transformers 5.x deprecatedconfig.use_return_dict, which silently returned tuples.- Expert dtype bridge —
w2(x.to(w2.weight.dtype))silently cast activations to int8 under weight-only quantization. - bf16 NaN — model runs fp32/fp16; no bf16 anywhere.
Limitations
- Tiny model: 110M params, most capacity in embeddings. Generations are often incoherent or factually wrong.
- Undertrained: 5K pretrain + 3K SFT steps (upstream). Production models train 100K+ steps on trillions of tokens.
License
Apache-2.0 (inherited from upstream neosantara/wader-100m).