wader-100m 🐟
A small ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following in Indonesian. Trained from scratch — no weights from DeepSeek-V4 were used.
Architecture
This model implements key DeepSeek-V4 innovations at a miniature scale:
| Component | Details |
|---|
| Parameters | ~110M total (41M embeddings, 69M non-embedding) |
| Hidden size | 320 |
| Layers | 8 |
| Attention heads | 8 (1 KV head — MQA-style) |
| MLA | Multi-head Latent Attention with q_lora_rank=160 |
| MoE | 4 routed experts + 1 shared, top-2 routing |
| Hyper-Connections | hc_mult=4, Sinkhorn routing (replacing residual connections) |
| MTP | 1 next-token prediction layer |
| Vocab | 129,280 (DeepSeek-V4 tokenizer) |
| Context | 2,048 tokens |
Training
Stage 1: Pretraining
- Dataset:
neosantara/IndoTalk (396k dialogs)
- Steps: 5,000 | Tokens: ~327M
- Batch: 32 effective | Seq length: 2,048
- Hardware: 1× NVIDIA RTX 5090 Blackwell
SFT Training Metrics
| Metric | Value |
|---|
| Final Train Loss | ~2.60 (Cross-Entropy at Step 3000) |
| Token Accuracy | ~51.71% (Predicting assistant responses) |
Stage 2: SFT (this model)
- Dataset:
neosantara/IndoTalkSFT (~160K conversations & instructions)
- Steps: 3,000 | Tokens: ~96M
- Batch: 16 effective | Seq length: 2,048
- Hardware: 1× NVIDIA RTX 5090 Blackwell
SFT Training Metrics
| Metric | Value |
|---|
| Final Train Loss | ~2.60 (Cross-Entropy at Step 3000) |
| Token Accuracy | ~51.71% (Predicting assistant responses) |
Usage
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
# IMPORTANT: This model requires a modern GPU (Ampere/Ada/Hopper) that supports bfloat16.
# Running on CPU or older GPUs (like T4) will result in NaN generation errors.
device = "cuda" if torch.cuda.is_available() else "cpu"
repo_id = "neosantara/wader-100m"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).to(device)
# Chat
messages = [{"role": "user", "content": "Halo, apa kabar?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)
outputs = model.generate(
input_ids,
max_new_tokens=200,
temperature=0.7,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
do_sample=True
)
print(tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True))
Limitations
- Tiny model: 110M params with 129K vocabulary — most capacity goes to embeddings. Generations are often incoherent or factually wrong.
- Undertrained: Only 5K pretrain + 3K SFT steps. Production models train for 100K+ steps on trillions of tokens.
- bf16 NaN: Use fp32 — the Hyper-Connections architecture produces values that overflow bf16 range at this scale.
- Custom code: Requires
trust_remote_code=True.