zagreus-0.4B-xmoons

Supervised fine-tune of mii-llm/zagreus-0.4B-ita for the mii-llm Post-Training Challenge, evaluated on ITALIC (10,000 Italian multiple-choice questions).

Result

Culture and commonsense 43.40 · Language capability 37.51 · unparsed answers 0.

The gain is statistically significant on the paired test set: McNemar chi2 = 326.4, p = 5.9e-73; paired bootstrap 95% CI of the delta [+10.84, +13.46]; Wilson 95% CI of the accuracy [40.06, 41.99].

Method

One supervised fine-tuning pass on gold answers — no distillation.

• Benchmark: the official italic.jsonl (10,000) and 5_shots.jsonl are fetched byte-for-byte from the ITALIC repository at a pinned commit and verified by SHA-256. Evaluation data is never rebuilt or modified.
• Data: Italian MCQ from four public HF datasets (revisions pinned), balanced by inverse frequency so language and culture are equally represented. The weighting derives from the data counts alone — no decision is taken by looking at the test set.
• Holdout: a dev split of 7,568 unique questions is carved before any upsampling and used for recipe selection; the ITALIC test is never used for tuning.
• Decontamination: two independent signals (TF-IDF char-ngram cosine >= 0.80 and MinHash/LSH word-shingle Jaccard >= 0.70) against both the test set and the official few-shot exemplars. Measured overlap of the final training pool: 0 with the 10,000 test questions, 0 with the 5-shot exemplars.
• Training: 1 epoch, loss on the completion only, max_len 2048 so the supervised target is never truncated, option permutation to remove the positional answer prior, few-shot demonstrations drawn from the training pool.
• Evaluation: the official fast protocol — templates, system message and extract_answer_fast reproduced verbatim from ITALIC's run_eval.py.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "icekern/zagreus-0.4B-xmoons"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {{"role": "system", "content": "Sei un assistente utile."}},
    {{"role": "user", "content": "Rispondi alla seguente domanda a scelta multipla ..."}},
]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

The model answers multiple-choice questions with a single letter, following the ITALIC fast prompt format.

Limitations

• The four weakest categories are all linguistic (morphology 30.71, synonyms 35.43, orthography 35.63, syntax 35.77). Public Italian MCQ corpora hold only ~2.5k unique grammar questions, so the balanced pool repeats them: the model sees little unique linguistic information.
• efederici/pinocchio and ITALIC draw on overlapping source material (Italian public-competition questions). Exact and near-duplicate contamination is removed and audited, but source-level overlap cannot be fully excluded.
• The reported score comes from a local HF-generate harness reproducing the official fast protocol.

Code

Full pipeline, raw evaluation output and per-example predictions: https://github.com/firekern/italic-post-training-challenge