zagreus-0.4B-italic-softkd
Submission for the mii-llm Italian Post-Training Challenge. This is mii-llm/zagreus-0.4B-ita (437M, Llama architecture) post-trained for the ITALIC benchmark, 10,000 multiple-choice questions on Italian culture and language.
| Model | ITALIC full 10K (5-shot fast, temp 0, official harness) |
|---|---|
| zagreus-0.4B-ita (base) | 0.2802 |
| public leaderboard best at submission time | 0.372 |
| this model | 0.4787 (confirmed 0.4784 on an independent rerun) |
Update (2026-08-05): superseded by idealab-cs2/zagreus-0.4B-italic-extkd — same recipe extended with two more softKD stages (culture pool, then external in-distribution pool), 0.4921/0.4929/0.4932 full-10K.
Training
One soft knowledge-distillation run from the base model, within the challenge budget of 10 H100 GPU hours (measured replay: ~4.9 H100h including data generation, teacher labeling and evaluation).
For every training question a 3-teacher committee (Mistral-Small-3.2-24B, Gemma-3-27B, Qwen3-32B, each fitting a single H100 in bf16) provides its probability distribution over the answer letters at the answer token. The student is trained in fp32 with a KL term on that distribution (temperature 2.0) plus cross-entropy on the gold letter (weight 0.5). The training pool has 21,606 items: pinocchio-derived knowledge MCQs, committee-labeled grammar MCQs, and generated orthography/morphology MCQs for the categories where public data is scarce. Every pool is decontaminated against the ITALIC test set (exact match plus semantic similarity at cosine 0.80; 0 exact leaks, max similarity 0.7999).
Training data is published on the Hub: italic-softkd-pool (the exact training file) (the exact training file; provenance splits knowledge/grammar/orthomorph included as named splits). Training code, the decontamination manifest and the complete report (including what did not work) are in the submission repository: https://github.com/ruggsea/italic-challenge-submission
Evaluation
Official ITALIC harness, unmodified: vLLM server, fast mode (letter-only answers), 5-shot, temperature 0, accuracy over all 10,000 questions. The chat template is embedded in the tokenizer config; serve the model with its own template.
vllm serve idealab-cs2/zagreus-0.4B-italic-softkd --chat-template-content-format auto
Per-category accuracy: civic education .56, lexicon .52, tourism .50, geography .47, art history .45, history .45, synonyms .47, syntax .42, orthography .39, morphology .36.
Acknowledgments
Training, data curation and evaluation by ruggsea, using compute resources of the Complex Social & Computational Systems (CS²) group, IDea_Lab, University of Graz.
Intended use
This model is specialized for Italian multiple-choice question answering in the ITALIC format. It is a 0.4B model: outside this format expect base-model-level capabilities.