Typakos-140M-it

The name "Typakos" comes from the Greek "Τυπάκος," meaning "small dude," a nod to the model's small (140M) parameter count.

Typakos-140M-it is the instruction-tuned, preference-aligned chat model built on top of alexliap/typakos-140m-base, a 140M-parameter bilingual (Greek/English) base language model. It's produced by a two-stage post-training pipeline on the base checkpoint: supervised fine-tuning (SFT) for instruction-following and chat formatting, followed by Direct Preference Optimization (DPO) for preference alignment. Both stages train all the parameters.

The full training pipeline, code, and configs live in scripts/typakos_140m/ on GitHub.

Model Details

Same architecture as the base model, with the vocabulary grown by 3 chat-template special tokens (added in the SFT stage, see below).

Tokenizer & Chat Template

Starts from the base model's byte-level BPE tokenizer (50,000 merges + 256 byte tokens + <|begin_of_text|>/<|end_of_text|>). The SFT stage adds three Llama-3-style special tokens (<|start_header_id|>, <|end_header_id|>, <|eot_id|>) and resizes the (tied) embedding matrix to match, with the new rows mean/covariance-initialized rather than random.

The chat template (adapted from TRL's llama3_training.jinja) renders each turn as:

<|start_header_id|>ROLE<|end_header_id|>

CONTENT<|eot_id|>

and wraps assistant turns in {% generation %} / {% endgeneration %} markers, which the SFT stage uses to mask the training loss to assistant tokens only (assistant_only_loss=True).

Training Pipeline

Stage 0: Pretraining

Trained from scratch, full-parameter, on ~9.33B tokens split roughly 50/50 English/Greek. See alexliap/typakos-140m-base for the full pretraining data mix and procedure.

Stage 1: Supervised Fine-Tuning (SFT)

Teaches the base checkpoint the chat format and instruction-following, via trl.SFTTrainer.

Data: alexliap/typakos_sft_dataset: 7 filtered subsets from 4 upstream Hub sources, kept only if well-formed (an optional single leading system turn, then alternating user/assistant turns ending on assistant) and token-bounded to fit the 2048-token context, then shuffled and split 90/10 train/validation.

~ 901M tokens total (~ 394M Greek, ~ 507M English), split into 936,491 train / 104,055 validation conversations (shuffle seed 0, 10% held out).

Stage 2: Direct Preference Optimization (DPO)

Aligns the SFT model to preference pairs via trl.DPOTrainer, starting from Stage 1's last checkpoint.

Data: openeurollm/Dolci-Instruct-DPO-translated (el + en configs): concatenates both language configs, shuffles (seed 0), and splits off 10% as validation. Rows are prompt/chosen/rejected conversational triples.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "alexliap/typakos-140m-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)

output = model.generate(input_ids, max_new_tokens=200, do_sample=False, temperature=0.7)
print(tokenizer.decode(output[0][input_ids.shape[-1] :], skip_special_tokens=True))

Example Outputs

Greedy-decoded (do_sample=False) chat completions from the final DPO checkpoint. Chosen to show both the model's capability and its main failure mode: without a repetition penalty, greedy decoding tends to lock onto a phrase and loop it.

English -- What is the capital of France?

The capital of France is Paris, which is the capital of France, and it is the capital of the French Empire. The capital is the capital of the French Empire, which is the capital of France. [...loops]

Greek -- Ποια είναι η πρωτεύουσα της Ελλάδας;

Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. Η πρωτεύουσα της Ελλάδας είναι η Αθήνα, η οποία είναι η πρωτεύουσα της Ελλάδας. [...loops]

Greek -- Τι είναι η τεχνητή νοημοσύνη;

Η τεχνητή νοημοσύνη (AI) είναι μια μορφή τεχνητής νοημοσύνης που χρησιμοποιείται για την ανάλυση και την ανάλυση δεδομένων. Είναι μια μορφή που χρησιμοποιείται για την ανάλυση δεδομένων, την ανάλυση δεδομένων και την ανάλυση δεδομένων. [...loops]

Both examples get the underlying fact right (Paris, Athens) and stay grammatical throughout, but repeat themselves rather than moving on once they lock onto a clause. Sampling avoids the repetition but trades it for occasional factual drift; see Limitations.

Evaluation

Evaluated 0-shot with EleutherAI lm-evaluation-harness, except BBH (3-shot) and GSM8K (5-shot). The model's own chat template was applied for IFEval; the remaining tasks were run as raw completion/multiple-choice scoring without the chat template, matching Open LLM Leaderboard-style methodology.

BBH here uses leaderboard_bbh, a multiple-choice reformulation, rather than the original free-form chain-of-thought task: a model this size cannot yet produce coherent multi-step reasoning text, so this number reflects loglikelihood-based answer ranking, not demonstrated step-by-step reasoning.

Greek Suite

Evaluated 0-shot on the ilsp_greek task suite (65 sub-tasks, including all 57 mmlu_greek subject splits), without the chat template. No SmolLM2 reference exists for these tasks, so they are reported standalone:

medical_mcqa_greek and mcqa_greek_asep have no English-benchmark equivalent and are reported for completeness only. winogrande_greek uses a different task formulation than the English winogrande row above (full-candidate-sentence plausibility scoring rather than fill-in-the-blank), so the two are not directly comparable.

Limitations

• Small (140M parameter) model; expect base-rate reasoning/knowledge limitations consistent with its scale and ~9.33B-token pretraining budget (see the base model card's evaluation table).
• Greedy decoding tends to loop once it locks onto a phrase (see Example Outputs); sampling avoids this at some cost to factual reliability.

License

MIT