Lewi Surge Preview 3B (Base)
An open mixture-of-experts language model for programming, trained from scratch by one person in Ukraine on free and donated compute.
This is the base model after code-focused pretraining and annealing. It is not instruction-tuned: use it for completion, few-shot prompting, or as a starting point for your own fine-tuning. For a chat checkpoint, see Lewi-Surge-Preview-3B-Chat-Early.
💛 This model is built without funding. The bottleneck is GPU time: about $3 buys one hour on an NVIDIA GH200. If you find Lewi useful or just want to see where it goes, you can support the next training stages on Ko-fi. Just $50 would unlock ~120 hours on a GH200: the provider gives us access once we make a $50 deposit, and our free credits cover the rest.
Model
| Parameters | ~3.45B total, ~0.5B active per token |
| Layers | 16 = 4 × (3 Gated DeltaNet linear attention + 1 full attention) |
| Experts | 128 per layer, top-6 routed + 1 shared, expert size 512 |
| Hidden size | 1024, 8 query / 2 KV heads |
| Vocabulary | 131,072 BPE + special tokens (Qwen3.6 vocabulary, pruned) |
| Context in training | 4,096 tokens (128k long-context stage is in progress) |
| Precision | bfloat16 |
Only 4 of 16 layers keep a KV cache (2 KV heads each), about 4 KB per token.
How it was trained
- Stage 1: a 1.4B model (12 layers, 64 experts) trained from scratch on ~3.2B tokens of FineWeb-Edu.
- Growth: grown to 3.45B without restarting: experts split 64 → 128, plus one more block of 4 layers.
- Code-first pretraining: ~1.6B tokens of curated code (with FIM), instructions, textbooks, math, and Ukrainian/Spanish/Portuguese/French/German web text.
- Annealing (this checkpoint): ~1.4B tokens, more than half conversations, learning rate decayed to zero.
JAX + MaxText, Muon optimizer. Hardware: rented/free H100/H200 sessions. Code: github.com/prostos2162-maker/lewi-surge.
Results
Before → after code-focused annealing (greedy decoding, standard prompts, local runs):
| Benchmark | Metric | Before | After (this model) |
|---|---|---|---|
| HumanEval | pass@1 | 18.9% | 25.0% |
| MBPP (sanitized) | pass@1 | 23.7% | 29.6% |
| HellaSwag | acc_norm | 41.9% | 43.4% |
| ARC-Challenge | acc_norm | 26.5% | 27.4% |
Knowledge-heavy benchmarks barely move at this size and token count.
Against small open base models
Same script for every model: base checkpoints, 0-shot, greedy; HumanEval uses the better of two standard prompt formats per model. Lewi is a work in progress; the others are final releases.
| Model | Training tokens | HumanEval | MBPP | ARC-C | HellaSwag |
|---|---|---|---|---|---|
| Qwen2.5-0.5B (Alibaba) | 18T | 29.9 | 37.7 | 32.3 | 49.5 |
| Qwen2.5-Coder-0.5B (Alibaba) | 23.5T | 27.4 | 48.2 | 30.2 | 47.3 |
| Lewi Surge Preview (this model) | 7.3B | 25.0 | 29.6 | 27.4 | 43.4 |
| SmolLM2-1.7B (Hugging Face) | 11T | 23.2 | 44.0 | 47.9 | 61.3 |
| Granite 3.0 1B-A400M (IBM) | 10T | 22.6 | 36.2 | 38.2 | 59.3 |
| Pythia-1.4B (EleutherAI) | 300B | 5.5 | 7.0 | 28.6 | 49.3 |
On code, Lewi lands between models trained on 1,400 to 3,200 times more data. On knowledge-heavy tasks it is behind, which is what more data fixes.
Usage
Requires transformers >= 5.0 (architecture Qwen3_5MoeForCausalLM).
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "PoliteChainsaw/Lewi-Surge-Preview-3B-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
prompt = 'def is_prime(n: int) -> bool:\n """Return True if n is a prime number."""\n'
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
Runs on a single consumer GPU (~7 GB in bf16).
Limitations
- A small model trained on well under 10B tokens in total: factual knowledge is weak and it can state wrong things confidently.
- Best at short Python functions; other languages and long programs are much weaker.
- Not safety-tuned. Do not use it for anything where mistakes matter.