Lewi Surge Preview 3B
A small open mixture-of-experts chat model, trained from scratch by one person in Ukraine on free and donated compute. This release replaces the earlier Chat-Early checkpoint with a full mid-training and chat fine-tuning run.
🧭 This is a transitional release. Lewi Surge Preview is a stepping stone toward the main model, Lewi Surge 24B-A2B. Its job is to prove the whole pipeline end to end: data, pre-training, mid-training, chat fine-tuning, export and release. With only ≈0.38B active parameters per token, it is not meant to compete with large assistants, so please judge it as what it is: a small, honest preview.
🔧 A good base for fine-tuning. Because so few parameters are active, fine-tuning is cheap: it fits on a single consumer GPU, and the model uses the standard Qwen3.5-MoE architecture, so it works in
transformersand vLLM without patches. Small models gain the most from task-specific fine-tuning, so if you have a narrow task, this is a good place to start. One tip from our own training: keep master weights in fp32 when fine-tuning. With bf16 weights and a small learning rate, most updates round away to zero.
💛 Support. Lewi is built without funding. If you'd like to see the 24B-A2B model happen, you can support it on Ko-fi.
What changed since Chat-Early
| Chat-Early | This release | |
|---|---|---|
| Mid-training | none | about 0.5B tokens of reasoning, math, code and science |
| Chat fine-tuning (SFT) | < 25% of the plan | full run, about 0.52B tokens |
| Identity | often claimed to be someone else | answers as Lewi Surge Preview, made in Ukraine by Polite Chainsaw |
| Ukrainian | rare | trained on about 16% Ukrainian conversations |
| Context | 16k | up to 128k (PoSE long-context stage; fine-tuned on 32k sequences) |
Known problems
| Problem | What you'll see |
|---|---|
| Weak Ukrainian | Ukrainian answers are grammatical but often shallow or repetitive. English is noticeably stronger. |
| Weak math | Arithmetic and multi-step math are often wrong (GSM8K below 2%). Don't use it for calculations. |
| Invented details about itself | It knows its name and maker, but may invent other details (founding year, team size and so on). |
| Factual errors | States wrong facts confidently, as expected at this size. |
| Repetition in long answers | Long poems or lists can loop. A light repetition_penalty (1.05–1.1) helps. |
| No thinking mode | The model was trained with a thinking mode, but at this size its reasoning traces often don't terminate. The released chat template turns thinking off. |
Most of these are limits of the model's size, not of its training. That's exactly why the next step is Lewi Surge 24B-A2B.
Benchmarks
Measured with lm-evaluation-harness 0.4.13, bf16. Multiple-choice tasks use the transformers backend on the released weights; generation tasks use greedy decoding on vLLM.
| Benchmark | Setting | Score |
|---|---|---|
| ARC-Easy | 0-shot, acc (first 2000 examples) | 54.2% |
| ARC-Challenge | 0-shot, acc_norm | 25.7% |
| HellaSwag | 0-shot, acc_norm (first 2000 examples) | 42.7% |
| GSM8K | 5-shot, flexible extract | 1.9% |
| HumanEval (instruct) | 0-shot, chat template | 29.3% |
| MBPP | 0-shot, chat template, code extracted from the reply (500 test tasks) | 19.0% |
| HumanEval | 0-shot, raw prompt (no chat template) | 8.5% |
| MBPP | 3-shot, raw prompt | 17.4% |
HumanEval with the chat template is the fair number for a chat model: without the template, the model tends to explain instead of writing code. For reference, the base model scored HumanEval 25.0% and MBPP 29.6% with our own evaluation script, so those two numbers are not directly comparable to this table.
Model
| Base | Lewi-Surge-Preview-3B-Base |
| Parameters | 3.45B total, ≈0.38B active per token (≈0.24B without embeddings) |
| Layers | 16 = 4 × (3 Gated DeltaNet + 1 full attention) |
| Experts | 128 per layer, top-6 routed + 1 shared, expert size 512 |
| Hidden size | 1024 |
| Vocabulary | 131,200 tokens |
| Context | up to 128k tokens: a long-context stage with PoSE on the base, then mid-training and SFT on 32k-token sequences. Needle-in-a-haystack at 128k: found in 2 of 3 runs |
| Chat format | ChatML (<|im_start|> / <|im_end|>), default system prompt built in, tools supported, thinking off |
| Training | mid-training and SFT in fp32 master weights on Google Cloud TPU v6e |
Usage
Requires transformers >= 5.0.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "PoliteChainsaw/Lewi-Surge-Preview-3B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Write a Python function that checks if a string is a palindrome."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.3, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Recommended settings: temperature 0.2–0.5. Runs on a single consumer GPU (about 7 GB in bf16). For fast generation, use vLLM: the transformers MoE implementation loops over experts and is much slower.
Training data
- Mid-training (about 0.5B tokens): Nemotron Post-Training v1 (math, STEM) and v2, OpenThoughts3 (code, math, science).
- SFT (about 0.52B tokens): Nemotron Post-Training v2 chat, math and code; SmolTalk2 (system chats, multi-turn instruction following, Aya, multilingual, tables, tool-calling traces); Tulu 3 persona instruction following; Nemotron tool use; safety data.
- Our own synthetic data: Ukrainian conversations, instruction following, reasoning and identity, generated with Apache-2.0 models (Qwen3.8-27B and gpt-oss).
- Not used: any Llama-generated data, or chat outputs from models whose licenses restrict training.
Limitations
Small model: expect wrong facts, weak math, uneven instruction following and weak Ukrainian. Not thoroughly safety-tuned. Don't use it for anything where mistakes matter.