Novi-Nano-Base
Novi-Nano-Base is a tiny causal language model trained from scratch by Novi-AI.
With just 1,258,560 parameters, Novi-Nano explores language modeling at an extremely small scale while remaining compatible with the Hugging Face Transformers ecosystem.
⚡ 1.26M parameters · 300M training tokens · 256-token context
Model Details
Architecture
| Property | Value |
|---|---|
| Model type | Causal Language Model |
| Parameters | 1,258,560 |
| Vocabulary size | 8,192 |
| Context length | 256 |
| Embedding size | 96 |
| Layers | 4 |
| Attention heads | 4 |
| FFN size | 384 |
| Tensor type | F32 |
Training
Novi-Nano-Base was trained from scratch using approximately 300 million training tokens.
Training Statistics
| Metric | Result |
|---|---|
| Training tokens | 300,023,808 |
| Best validation loss | 5.418699 |
| Final validation loss | 5.418699 |
| Final validation perplexity | 225.5853 |
Tokenizer
Novi-Nano uses a custom tokenizer with a vocabulary size of 8,192 tokens.
The tokenizer was trained using data from:
- FineWeb-Edu
- FineWeb-HQ
- SmolLM-Cosmopedia
Intended Use
Novi-Nano-Base is primarily intended for:
- 🔬 Research and experimentation
- 🧪 Small-model language-model experiments
- 🎓 Educational purposes
- 🛠️ Fine-tuning experiments
- 💻 Lightweight local inference
As a base model, it is not specifically instruction-tuned for following user commands or acting as a conversational assistant.
Limitations
Novi-Nano-Base is an extremely small experimental language model.
Because of its size and short context window, it will have significant limitations compared with modern billion-parameter language models.
It may:
- Generate incoherent text
- Repeat phrases
- Produce factual errors
- Struggle with complex instructions
- Have limited world knowledge
- Perform poorly on reasoning tasks
- Lose context beyond its 256-token window
This model should be considered a research and experimentation model, rather than a production-ready general-purpose LLM.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Novi-AI/Novi-Nano-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "Hello, my name is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=50,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Project History
Novi AI follows the earlier AppleMind experiments, with Novi becoming the primary project for developing small language models.
AppleMind → Novi AI → Novi-Nano 🚀
Acknowledgements
Novi-Nano was built using the open-source machine-learning ecosystem and datasets made available by the community.
Special thanks to:
- Hugging Face 🤗
- FineWeb
- SmolLM
- Cosmopedia
License
This model is released under the Apache 2.0 license.
🧠 Novi AI
Small models. Big experiments.
Novi-Nano is intentionally tiny — exploring how far a language model can go with just a fraction of the parameters used by modern LLMs.
Novi AI 2026 — Project Kairo