Lil Bard 172M MoE
Lil Bard is a small English story language model pretrained from scratch. It is a base model, not an instruction-tuned or chat model.
The model has 172,052,992 total parameters and 58,806,784 active parameters per token. It uses 16 transformer layers, width 512, 8 feed-forward experts with top-2 routing, and a maximum exported context length of 32,768 tokens.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "N8Programs/lil-bard"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.8,
top_p=0.95,
pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The tokenizer automatically prepends BOS. Its special-token IDs are EOS 0, BOS 8190, and PAD 8191.
Architecture
| Property | Value |
|---|---|
| Total parameters | 172,052,992 |
| Active parameters/token | 58,806,784 |
| Layers | 16 |
| Hidden size | 512 |
| Attention heads / KV heads | 4 / 2 |
| Head dimension | 128 |
| Experts / selected experts | 8 / 2 |
| Dense MLP size | 1,536 |
| Expert MLP size | 768 |
| Vocabulary | 8,192 |
| Maximum exported context | 32,768 |
| Published weight dtype | BF16 |
The checkpoint uses the stock Transformers Qwen3MoeForCausalLM layout. MoE
expert weights are stored as per-expert gate_proj, up_proj, and down_proj
tensors for compatibility across Transformers releases; loading has been tested
with Transformers 4.57.1 and 5.11.0.
Tokenizer
The 8,192-entry tokenizer is a byte-level BPE tokenizer trained on a balanced 1.5-million-document sample of the corpus. It does not use regex, whitespace, or word pretokenization. BOS, EOS, PAD, and UNK are distinct tokens.
Training data
The corpus contained 8,732,634 documents drawn from:
klusai/ds-tf1-en-3mkarpathy/tinystories-gpt4-clean- A deterministic 3-million-row sample of
littlelearner/LittleCurriculum
A canonical validation set excluded 1,000 DS-TF1 test rows and 1,000 TinyStories test rows from training.
The model trained for exactly 2,492,032,616 real loss tokens over 25,485
distributed steps on two NVIDIA GB10 systems. Whole-document packing achieved
99.4713% utilization. Training used a local adaptation of
N8python/spark-gpt.
The complete training trace is available in the
lil_bard_moe_8x2_full W&B run.
Evaluation
| Evaluation | Result |
|---|---|
| Canonical validation loss | 1.42228 nats/token |
| ARC-Easy zero-shot accuracy | 32.15% |
| ARC-Easy zero-shot normalized accuracy | 32.79% |
ARC-Easy was evaluated on all 2,376 test questions with lm-eval 0.4.12 in BF16, using the base-model prompt format and an explicit BOS token.
Historical checkpoints
To keep ordinary downloads of this repository small, the 25 periodic
checkpoints are published separately in
N8Programs/lil-bard-checkpts.
They span step 1,000 through step 25,000 in increments of 1,000.
Limitations
This model was trained primarily on simple synthetic stories. It has limited world knowledge and reasoning ability, may produce repetitive or incoherent text, and has not been safety-tuned. Do not use it for factual, medical, legal, financial, or other high-stakes decisions.