ShallowSeek-mini-base

ShallowSeek-mini-base is a tiny DeepSeek-V3-style Mixture-of-Experts language model trained from scratch on a laptop CPU. It has 11.0M parameters in total, of which 6.3M are active per token. It was pretrained on 286M tokens of FineWeb-Edu.

Part of the ShallowSeek family. Not affiliated with DeepSeek. The name is a nod to the architecture it borrows, at a fraction of the depth.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows very few facts and confidently makes things up. Don't use it for anything that matters.

Architecture

It follows the main ideas of DeepSeek-V3, scaled down:

Training

Validation loss: 5.12 (step 1,000) → 4.28 (step 5,000) → 4.06 (step 10,000) → 3.97 (step 15,000) → 3.87 (step 20,000) → 3.78 (step 25,000) → 3.70 (step 30,000) → 3.67 (step 35,000)

Evaluation

Factual probes (benchmark.py)

These are 14 fill-in-the-blank probes and 12 true-versus-false sentence pairs, scored on raw text.

MMLU (mmlu_eval.py)

By category (accuracy weighted by question count; chance is 25%):

Biology & medicine (cloze): 27.8% on 2,094 questions, against 25% chance. That's +2.8 points, about 3.0 standard errors above chance, so it's statistically significant. 9 of the 10 subjects are at or above chance. The overall cloze score is close to chance, so the signal sits mostly in the life-science and health subjects, where FineWeb-Edu's educational web text is richest. Maths-heavy subjects come out below chance, since cloze can't guess numeric answers. This group isn't an official MMLU category; it pools these 10 subjects:

Chance is 25%. In the letter format the model compares " A", " B", " C" and " D", with up to 5 same-subject examples included only while they fit in the 1,024-token window. In the cloze format each answer's text is scored by log-probability per byte. Small models in the letter format tend to answer the same letter within a subject, so per-subject letter scores mostly reflect how the answer key happens to be distributed. The cloze results are the meaningful ones here.

What it can and can't do

• ✅ Grammatical, topic-consistent English for a few sentences, in the register of educational web text.
• ✅ Coarse associations: it knows HIV is a virus rather than a mineral, ice is frozen water, and a week has seven days.
• ❌ Specific facts and relations (capital cities, which body orbits which), arithmetic beyond memorised cases like "2 + 2 = 4", and following instructions.
• ❌ It invents names, numbers and "facts" fluently, and falls into repetition loops at low temperature. Use top-p ≈ 0.9 and a repetition penalty of about 1.1–1.2.

Usage

llama.cpp, Ollama or LM Studio (GGUF, llama.cpp's built-in deepseek2 architecture):

llama-completion -m ShallowSeek-mini-base-q8_0.gguf -p "Photosynthesis is" -n 100 --temp 0.7 --top-p 0.9 --repeat-penalty 1.15

PyTorch (needs the deepseek_moe package from the training code):

import json, torch
from safetensors.torch import load_file
from deepseek_moe import DeepSeekMoEModel, ModelConfig
from deepseek_moe.tokenizer import BPETokenizer

model = DeepSeekMoEModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Files

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE, auxiliary-loss-free balancing and multi-token prediction.
• Pretraining data: FineWeb-Edu (Hugging Face, ODC-BY 1.0).

Card generated 2026-10-01 by prepare_release.py.