Nesso-0.4B-Instruct

Nesso-0.4B-Instruct is a bilingual English/Italian Small Language Model (SLM) optimized for conversational and instruction-following use cases. It is post-trained on top of Zagreus-0.4B-ita, a foundational model trained from scratch by the mii-llm community (Made in Italy – Large Language Model) on the Seeweb HPC infrastructure.

Designed for sovereign edge inference, Nesso-0.4B-Instruct delivers competitive instruction-following performance in both Italian and English at a fraction of the compute cost of larger models.

⚠️ This model is currently at the SFT (Supervised Fine-Tuning) stage. DPO (Direct Preference Optimization) training is planned and updated results will be published upon completion.

Model Details

Training Details

Base Model Pre-training

Nesso-0.4B-Instruct is built on Zagreus-0.4B-ita, which was pre-trained on approximately 1 trillion tokens using the following data mix:

Token distribution: ~400B English + ~400B Italian + ~200B Code
Infrastructure: 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs) on Seeweb HPC
Framework: Nanotron (mii-llm fork)

Post-training (SFT)

Post-training was performed using Axolotl with FSDP across 4 nodes (32× A100 GPUs).

The instruction dataset is a proprietary bilingual (English/Italian) corpus curated by the mii-llm team, with long-term iteration across domains including instruction following, conversational AI, and general knowledge. This dataset is considered a strategic research asset and is not released as open source.

Key hyperparameters:

Chat Template

This model uses the ChatML format:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Ciao! Come stai?<|im_end|>
<|im_start|>assistant

Special tokens:

• pad_token: <|im_end|>
• eos_token: <|im_end|>

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mii-llm/nesso-0.4B-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "Sei un assistente utile e preciso."},
    {"role": "user", "content": "Spiegami cos'è un modello linguistico di grandi dimensioni."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

Evaluation

Evaluation Commands

# Italian benchmarks
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size 1

lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks hellaswag_it,arc_it --device cuda:0 --batch_size 1

lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks ifeval-ita --device cuda:0 --batch_size 1

# English benchmarks
lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks mmlu --num_fewshot 5 --device cuda:0 --batch_size 1

lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks hellaswag,arc --device cuda:0 --batch_size 1

lm-eval --model hf --model_args pretrained=mii-llm/nesso-0.4B-instruct \
  --tasks ifeval --device cuda:0 --batch_size 1

Results

English Benchmarks

Italian Benchmarks

Overall

Discussion

Nesso-0.4B-Instruct achieves the highest IFEval English score (0.3465) among all compared models — including the larger Qwen3-0.6B — demonstrating strong instruction-following capability. On Italian HellaSwag, it also leads with 0.4076.

Qwen3-0.6B maintains a clear advantage on MMLU in both languages. MMLU is a widely used benchmark that is frequently represented in training corpora; we believe our results nonetheless demonstrate a highly competitive SLM for English/Italian edge deployment scenarios.

Related Models

Citation

If you use this model in your research, please cite:

@misc{nesso2025,
  title        = {The Joy and Pain of Training an LLM from Scratch:
                  A Technical Report on the Zagreus and Nesso Model Families},
  author       = {mii-llm community},
  year         = {2025},
  howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
}

Acknowledgements

• Antonio Baldassarra (CEO, Seeweb) and Marco Cristofanilli (Head of AI, Seeweb) for infrastructure sponsorship
• The Hugging Face team for Nanotron, datatrove, FineWeb, and FineWeb-2
• The mii-llm open-source community

License

Released under the Apache 2.0 license.

Made with ❤️ in Italy by mii-llm