Sai-Slice1-60M-base

Sai-Slice1-60M-base is a small Mixture-of-Experts (MoE) decoder-only language model, pretrained from scratch on ~1.2B tokens of English web text. It is the first release (slice 1) of the Sai series.

This is a base model: it continues text and has not been instruction-tuned. A chat fine-tune (Sai-Slice1-60M-Chatter) is coming soon.

Model details

Each token is routed to 4 of the 24 routed experts, and always passes through the shared expert. The first layer uses a dense FFN; the other 14 use MoE. Routing is trained with a load-balancing auxiliary loss.

Parameter breakdown: token embeddings 4.19M (tied with the output head), attention 3.93M, dense FFN 0.79M, MoE layers 51.7M (14 × 25 experts × 147K + routers). Active per token counts 5 of the 25 experts in each MoE layer.

Usage

The model uses custom code, so trust_remote_code=True is required. It was built and tested with transformers==5.16.1 and torch==2.11.0.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "OliverSundaram/Sai-Slice1-60M-base"

model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

text, tokens_per_sec, time_to_first_token = model.run(
    "The history of the printing press",
    tokenizer,
    device,
    max_new_tokens=100,
    top_k=20,
    temp=0.9,
    use_cached=True,
    format_prompt=False,  # plain text continuation for the base model
)
print(text)

model.run returns the prompt plus its continuation, the generation speed in tokens/sec, and the time to first token in seconds. Generation stops at <|end|>, at max_new_tokens, or at the 1,024-token context limit.

Notes:

• Keep format_prompt=False with this base model. format_prompt=True wraps the prompt in the chat template (<|system|>…<|end|><|user|>…<|end|><|assistant|>), which is intended for the upcoming fine-tuned model.
• Use model.run for generation. The Hugging Face generate() method is not supported, because the model manages its own KV cache instead of returning past_key_values.
• A standard forward pass (model(input_ids, attention_mask=...)) returns logits and works with tools such as lm-evaluation-harness.

Training data

Tokenizer: byte-level BPE trained on the first 1M documents of the same subset. Vocabulary size is 16,384, maximum token length is 16, and there are five special tokens: <|end|> (EOS), <|pad|>, <|user|>, <|assistant|> and <|system|>.

Training procedure

Final results: test cross-entropy 2.101 (perplexity ≈ 8.2), test auxiliary loss 1.033.

Evaluation

Evaluated with lm-evaluation-harness v0.4.13 on the full test sets.

MMLU by category: humanities 24.3, STEM 21.9, social sciences 21.7, other 24.2.

Inference speed

Measured with model.run on 100 prompts from gharezlak/autocomplete, generating up to 100 new tokens (top-k 20, temperature 0.9, KV cache on), in float32 on an RTX 4060 (8 GB).

The first prompt includes CUDA warmup (1.85 s time to first token). Throughput is currently limited by the MoE layer, which loops over experts in Python, rather than by model size.

Limitations

• Base model only. It continues text and does not follow instructions or hold a conversation.
• Small. With ~60M parameters and ~1.2B training tokens, it has very limited world knowledge and reasoning, and it often produces fluent but incorrect or incoherent text. Do not rely on its outputs for facts.
• English only, with a 1,024-token context window.
• Data bias. It was trained on filtered and synthetically rewritten web text and may reproduce the biases and errors in that data.
• It is intended for research, education and experimentation, not for production use.

Files

Citation

@misc{sundaram2026sai,
  title        = {Sai-Slice1-60M-base: A Small Mixture-of-Experts Language Model},
  author       = {Sundaram, Oliver},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/OliverSundaram/Sai-Slice1-60M-base}}
}

License

Released under the Apache 2.0 License.