🧠 Nano-Nano v5.1

~1218.3 M · Qwen3 · 300M · GQA + QK-Norm · Sequence-Packed · 26 Datasets

Fully redesigned successor to Nano-nano v4.5.
~298M Qwen3 parameters trained with sequence packing on a quality-tiered 34-dataset mix. Features loss-boost system: auto-extends training if loss > 4.95 (up to 3×75 steps).
Goal: loss < 2.5 through compute efficiency, not raw scale.

📋 Summary

🏗️ Architecture (v4 → v4.5 → v5.1)

📊 Evaluation

Hallucination resistance tests whether the model correctly declines or hedges on unanswerable questions (future events, fictional entities, impossible premises).

🍳 Training

What's new in v5.1

Settings

Dataset Mix (21 datasets, quality-tiered)

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/Nano-Nano_v5.1", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/Nano-Nano_v5.1")

def chat(prompt: str, max_new_tokens: int = 256) -> str:
    # <think> opens the reasoning block; model outputs reasoning then </think> then answer
    text = ("<|im_start|>user
" + prompt + "<|im_end|>
"
            "<|im_start|>assistant
<think>
")
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(
        **inputs, max_new_tokens=max_new_tokens,
        do_sample=True, temperature=0.7, top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:],
                            skip_special_tokens=True).strip()

print(chat("Write a Python function to merge two sorted lists."))
print(chat("Solve: if 3x + 7 = 22, what is x?"))
print(chat("Explain transformer attention in simple terms."))

⚠️ Limitations

• Context limited to 2 048 tokens
• Trained on 0.01 B tokens — far below production scale
• Pascal GPU (GTX 1080): fp16 AMP only, no bf16
• Not RLHF/DPO aligned

📜 Citation

@misc{nano-nano-v5,
  author       = {ray0rf1re},
  title        = {Nano-Nano v5.1: 300M LLaMA with Sequence Packing},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/Nano-Nano_v5.1},
}