⚛ Quark-50M-v2
43.8M parameter Italian-first bilingual language model, trained from scratch by ThingAI.
Quark-50M-v2 is an ultra-compact causal language model that speaks fluent Italian. Designed as a proof-of-concept for small, efficient, Italian-centric AI.
Highlights
- 43.8M parameters — runs on any device, even CPU
- Italian-first — trained on 60% Italian data (books, Wikipedia, web)
- ChatML format —
<|im_start|>user/<|im_start|>assistant - Custom tokenizer — 16k BPE, optimized for Italian (4.15 chars/token)
- Trained from scratch — architecture, tokenizer, and training pipeline all custom
Architecture
| Component | Value |
|---|---|
| Parameters | 43.8M |
| Vocabulary | 16,384 (BPE) |
| Dimensions | 512 |
| Layers | 12 |
| Heads | 8 (4 KV heads, GQA) |
| FFN | 1,408 (SwiGLU) |
| Context | 2,048 tokens |
| Normalization | RMSNorm |
| Position | RoPE |
| Weight Tying | Yes |
Training
Pretraining: 5B tokens on a curated mix:
| Dataset | Weight | Type |
|---|---|---|
| PleIAs/Italian-PD | 25% | 171K Italian books (public domain) |
| FineWeb-2 Italian | 20% | Cleaned, deduplicated web |
| Wikipedia IT | 15% | Encyclopedia |
| Cosmopedia | 15% | Synthetic educational |
| SmolLM-Corpus | 10% | Curated mix |
| StarCoder Python | 8% | Code |
| OpenWebMath | 7% | Mathematics |
SFT: Fine-tuned on quattro-chiacchiere, a synthetic Italian Q&A dataset generated with Alembic.
Usage
import torch
from huggingface_hub import hf_hub_download
from transformers import PreTrainedTokenizerFast
# Load
ckpt_path = hf_hub_download("ThingAI/Quark-50M-v2", "model.pt")
model_py = hf_hub_download("ThingAI/Quark-50M-v2", "model.py")
tok_file = hf_hub_download("ThingAI/Quark-50M-v2", "tokenizer.json")
# Tokenizer
tokenizer = PreTrainedTokenizerFast(tokenizer_file=tok_file)
tokenizer.eos_token = "<|endoftext|>"
# Model
import importlib.util
spec = importlib.util.spec_from_file_location("model", model_py)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
cfg = mod.ModelConfig(**ckpt["model_cfg"])
model = mod.Quark(cfg).eval()
model.load_state_dict(ckpt["model"])
# Chat
prompt = "<|im_start|>user\nQual è la capitale d'Italia?<|im_end|>\n<|im_start|>assistant\n"
ids = tokenizer.encode(prompt, return_tensors="pt")
with torch.no_grad():
for _ in range(100):
logits = model(ids)[1][:, -1, :].float()
nxt = logits.argmax(-1, keepdim=True)
if nxt.item() == tokenizer.convert_tokens_to_ids("<|im_end|>"): break
ids = torch.cat([ids, nxt], -1)
print(tokenizer.decode(ids[0], skip_special_tokens=True))
# La capitale d'Italia è Roma.
Examples
Tu: Qual è la capitale d'Italia?
Quark: La capitale d'Italia è Roma.
Tu: Chi sei?
Quark: Sono Quark, piacere di conoscerti.
Tu: Come ti chiami?
Quark: Mi chiamo Quark, piacere di conoscerti.
Limitations
- 43.8M parameters — cannot perform complex reasoning or long-form generation
- Factual accuracy — may hallucinate facts, especially on niche topics
- SFT dataset — currently limited; more data will improve reliability
- No safety training — not recommended for production without guardrails
Related
- Quark3Tokenizer — the tokenizer
- quattro-chiacchiere — the SFT dataset
- Alembic — the dataset distillation tool
- Glyph — multi-task text classifier by ThingAI
Citation
@misc{quark50m,
author = {ThingAI},
title = {Quark-50M-v2: Italian-First Small Language Model},
year = {2026},
url = {https://huggingface.co/ThingAI/Quark-50M}
}
License
Apache 2.0