🧠 Nano-nano v4.7

~296M · Qwen3-style · Custom BPE Tokenizer · ChatML + Thinking · 43 Datasets

Successor to Nano-nano v4.6 — redesigned with a custom corpus-trained BPE tokenizer, native thinking / chain-of-thought support, and a quality-tiered 43-dataset mix with sequence packing for 100% token efficiency. (currently undertrained!)

📋 Summary

🏗️ Architecture

Qwen3-style decoder with GQA and QK-Norm, scaled for ~296 M parameters with a 32k-range custom tokenizer.

🧩 Custom Tokenizer

Nano-nano v4.7 ships with a byte-level BPE tokenizer trained on the actual training corpus.

• Vocab size: 49 664 (minimum 49 529, padded to ×128)
• Byte-level: zero <unk> tokens — every unicode character is representable
• ChatML specials baked in (not added after): <unk> <s> </s> <pad> <|im_start|> <|im_end|> <|system|> <|user|> <|assistant|> <think> </think>
• Jinja2 chat template set for apply_chat_template() compatibility

Load with:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ray0rf1re/nano-nano_4.7")

💭 Thinking / Chain-of-Thought

v4.7 is the first Nano-nano model with native thinking support.

The <think> and </think> tokens are part of the tokenizer vocabulary from the start (indices 9 & 10), so BPE never splits them.

Generation format:

<|im_start|>user
What is 17 × 13?<|im_end|>
<|im_start|>assistant
<think>
17 × 13 = 17 × 10 + 17 × 3 = 170 + 51 = 221
</think>
221<|im_end|>

Inference frameworks that open with <|im_start|>assistant\n<think>\n will prompt the model to reason before answering.

🚀 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/nano-nano_4.7",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/nano-nano_4.7")

def chat(prompt: str, think: bool = True, max_new_tokens: int = 512) -> str:
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True)
    if think:
        text += "<think>\n"   # open reasoning block
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(
        **inputs,
        max_new_tokens     = max_new_tokens,
        do_sample          = True,
        temperature        = 0.7,
        top_p              = 0.9,
        repetition_penalty = 1.1,
        pad_token_id       = tokenizer.eos_token_id,
    )
    new_ids = out[0][inputs["input_ids"].shape[-1]:]
    return tokenizer.decode(new_ids, skip_special_tokens=False).strip()

# With thinking
print(chat("Solve: if 3x + 7 = 22 what is x?"))

# Without thinking
print(chat("Write a haiku about coding.", think=False))

🍳 Training

Dataset Mix (43 datasets, quality-tiered)

Settings

⚠️ Limitations

• Context limited to 1 024 tokens during training (2 048 at inference)
• Pascal GPU (GTX 1080): fp16 AMP only, no bf16
• Not RLHF/DPO aligned — outputs may vary in safety and tone
• Thinking quality proportional to training data quality

📜 Citation

@misc{nano-nano-47,
  author       = {ray0rf1re},
  title        = {Nano-nano v4.7: Qwen3-style LM with Custom Tokenizer and Thinking},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/nano-nano_4.7},
}

🛰️ RSOSTBTEST-Plus Results (2026-06-29)

Overall: 4.4% (9/250 tasks passed) — full results