Typakos-140M-base
The name "Typakos" comes from the Greek "Τυπάκος," meaning "small dude," a nod to the model's small (140M) parameter count.
Typakos-140M-base is a 140M-parameter bilingual (Greek/English) base language model with a Llama-family architecture and a 2048-token context length. It was trained from scratch (no warm start), full-parameter, on approximately 9.33B tokens split roughly 50/50 between English and Greek. This is a base, completion-only checkpoint: it has no chat template and has not undergone any instruction tuning, so it should be prompted as a plain text continuation model, not as a chat assistant.
The full training pipeline, code, and configs live in scripts/typakos_140m/ on GitHub.
Model Details
| Model Configuration | Value |
|---|---|
| Layers | 18 |
| Hidden size | 768 |
| Intermediate size | 1792 |
| Attention heads | 12 (query) / 6 (KV, grouped-query attention) |
| Head dim | 64 |
| Context length | 2048 |
| Vocab size | 50,258 |
| Tied embeddings | yes |
| Precision | bf16 |
| Parameters | ~145M total, ~106M non-embedding |
Tokenizer
A byte-level BPE tokenizer with 50,000 learned merges plus 256 byte tokens and 2 special tokens (<|begin_of_text|>, id 0; <|end_of_text|>, id 1, which also doubles as the pad token). The pre-tokenizer regex pipeline is structurally borrowed from Llama 3.2, but the vocabulary itself was trained from scratch on the bilingual corpus below.
The tokenizer's post-processor wraps every encode in <|begin_of_text|> ... <|end_of_text|> by default. This is convenient for training but wrong for a generation prompt (a trailing EOS tells the model the prompt is already a finished document). See the usage example below for how to bypass it.
Training Data
Sourced with a 50/50 English/Greek token target:
| Source | Repo | Language |
|---|---|---|
| fineweb_edu | HuggingFaceFW/fineweb-edu | en |
| fineweb_hq_el | alexliap/high-quality-gr-text | el |
| finewiki_el | alexliap/high-quality-gr-text | el |
| wikipedia_el | alexliap/high-quality-gr-text | el |
| synth_faq | alexliap/greek-synth-v1 | el |
| synth_math | alexliap/greek-synth-v1 | el |
| synth_table | alexliap/greek-synth-v1 | el |
| synth_tutorial | alexliap/greek-synth-v1 | el |
Training Procedure
| Training Configuration | Value |
|---|---|
| GPUs | 8 |
| Batch size | 16 sequences/GPU x 2048 tokens = 262,144 tokens/step |
| Gradient accumulation | 1 |
| Optimizer | AdamW, lr 2e-4, betas (0.9, 0.95), eps 1e-10, weight_decay 0.001 |
| LR schedule | Linear warmup, 2000 steps |
| Tokens seen | ~9.33B (1 epoch, run stopped at corpus exhaustion) |
| Precision | bf16 |
Reproduction
All code, Hydra configs, and step-by-step reproduction commands (data download, tokenizer training, shard packing, and the training launch script) are in scripts/typakos_140m/ in the pretrain GitHub repo, under the README.md there:
- Environment setup and initial data pull (
setup.sh) - Train the bilingual BPE tokenizer (
train_tokenizer.py) - Measure bytes/token per source and derive the token mix plan (
measure_token_rates.py) - Download the full corpus per that plan (
download_data.py) - Mix sources into shuffled, proportional shards (
concat_data.py) - Tokenize and pack into 2048-token sequences (
prepare_shards.py) - Train (
train.sh, pointed atconfigs/train.yaml)
How to Use
This is a base model, so prompt it with plain text and let it continue. Because the tokenizer auto-wraps every encode in BOS/EOS, use add_special_tokens=False and prepend BOS manually to avoid signaling a "finished document" to the model:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "alexliap/typakos-140m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The history of Athens begins"
input_ids = tokenizer(
tokenizer.bos_token + prompt, add_special_tokens=False, return_tensors="pt"
).input_ids
output = model.generate(input_ids, max_new_tokens=100, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Evaluation
Evaluated 0-shot with EleutherAI lm-evaluation-harness, except GSM8K at 5-shot, matching the shot conventions stated on the SmolLM2 model card.
| Benchmark | Typakos-140M-base | SmolLM2-135M-8k |
|---|---|---|
| HellaSwag (acc_norm) | 26.5 | 42.1 |
| ARC (avg acc_norm) | 28.3 | 43.9 |
| PIQA (acc_norm) | 56.6 | 68.4 |
| MMLU cloze (acc) | 23.4 | 31.5 |
| CommonsenseQA (acc) | 19.6 | 33.9 |
| TriviaQA (exact_match) | 0.0 | 4.1 |
| Winogrande (acc) | 49.6 | 51.3 |
| OpenBookQA (acc_norm) | 33.2 | 34.6 |
| GSM8K 5-shot (exact_match) | 0.2 | 1.4 |
The gap to SmolLM2-135M-8k below is largely a training-budget gap, not an architecture gap: per its model card, SmolLM2-135M was pretrained on about 2T tokens, roughly 200 times the ~9.33B tokens Typakos-140M-base has seen, on a curated English-only mix (FineWeb-Edu, DCLM, The Stack, plus additional filtered sources). Typakos-140M-base is a single-stage, from-scratch run on a much smaller, bilingual (EN/EL) corpus, so it is comparable in size and architecture but not yet in data scale.
Greek Suite
Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits). No SmolLM2 reference exists for these tasks, so they are reported standalone:
| Task | Typakos-140M-base |
|---|---|
| arc_challenge_greek (acc) | 20.2 |
| hellaswag_greek (acc) | 26.5 |
| mmlu_greek (acc, 57 subj.) | 23.0 |
| mgsm_direct_greek (exact_match) | 3.6 |
| winogrande_greek (acc) | 49.7 |
| truthfulqa_greek_mc1 (acc) | 23.9 |
| truthfulqa_greek_mc2 (acc) | 44.5 |
| medical_mcqa_greek (acc) | 15.3 |
| mcqa_greek_asep (acc) | 18.3 |
medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.
Limitations
- It is a base, completion-only model with no instruction tuning or chat capability.
- It is best suited for continued pretraining, fine-tuning, or research experimentation, not for direct deployment as an assistant.
License
MIT