Novi-Micro-Base
Novi-Micro-Base is a tiny causal language model trained from scratch by Novi-AI.
With approximately 5.04 million parameters, Novi-Micro explores language modeling at a small scale while using a substantially larger context window and training corpus than earlier Novi models.
⚡ 5.04M parameters · 1B training tokens · 2,048-token context
Model Details
Architecture
Novi-Micro-Base uses a custom BananaMind 2-style decoder architecture with RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, QK normalization, SwiGLU feed-forward layers, and tied input/output embeddings.
| Property | Value |
|---|---|
| Model type | Causal Language Model |
| Architecture | BananaMind 2-style |
| Parameters | 5.04M |
| Vocabulary size | 16,384 |
| Context length | 2,048 tokens |
| Attention | Grouped-Query Attention (GQA) |
| Position encoding | RoPE |
| Normalization | RMSNorm |
| Feed-forward | SwiGLU |
| QK normalization | Enabled |
| Embeddings | Tied |
| Training precision | FP32 |
Training
Novi-Micro-Base was trained from scratch for approximately 1 billion tokens.
The training configuration used a 5M parameter target, a sequence length of 2,048 tokens, and a single dataset source.
Training Configuration
| Setting | Value |
|---|---|
| Training mode | Pretraining |
| Target size | 5M parameters |
| Actual parameters | 5.04M |
| Training tokens | 1,000,000,000 |
| Sequence length | 2,048 |
| Batch size | 2 |
| Gradient accumulation | 8 |
| Effective batch size | 16 sequences |
| Learning rate | 2 × 10⁻⁴ |
| Scheduler | Cosine |
| Warmup ratio | 3% |
| Weight decay | 0.01 |
| Maximum gradient norm | 1.0 |
| Optimizer steps | 30,518 |
| Precision | FP32 |
| Torch compile | Enabled |
| Random seed | 1337 |
| Checkpoint interval | 100 steps |
| Logging interval | 10 steps |
The model was trained until the selected 1,000,000,000-token target was consumed.
Dataset
Novi-Micro-Base was trained using:
- FineWeb-HQ
The training configuration allocated a token target of exactly:
1,000,000,000 tokens
Documents were streamed from the dataset and packed into contiguous 2,048-token sequences before being passed to the model.
Tokenizer
Novi-Micro uses a custom byte-level BPE tokenizer with a vocabulary size of 16,384 tokens.
The tokenizer was trained from samples drawn from the training dataset before model pretraining.
Special tokens include:
[PAD][BOS][EOS][UNK]
The tokenizer uses ByteLevel pre-tokenization and decoding.
Architecture Details
Novi-Micro uses a decoder-only Transformer architecture.
Attention
The model uses grouped-query attention (GQA), where multiple query heads share key and value heads.
Rotary Position Embeddings (RoPE) are applied to the query and key representations.
The BananaMind 2-style architecture also applies RMSNorm to the query and key head dimensions.
Feed-Forward Network
Each Transformer block uses a SwiGLU feed-forward network:
SwiGLU(x) = SiLU(gate(x)) × up(x)
The result is projected back to the model's hidden dimension.
Normalization
The model uses RMSNorm before both the attention and feed-forward sublayers.
Embeddings
The input token embeddings and language-model output embeddings are tied, reducing the number of independent parameters.
Intended Use
Novi-Micro-Base is primarily intended for:
- 🔬 Research and experimentation
- 🧪 Small-model language-model experiments
- 🎓 Educational purposes
- 🛠️ Fine-tuning experiments
- 💻 Lightweight local inference
- 🤖 Exploring language modeling at the million-parameter scale
As a base model, Novi-Micro-Base is not instruction-tuned and is not specifically trained to follow user commands or behave as a conversational assistant.
Limitations
Novi-Micro-Base is a very small experimental language model.
With approximately 5 million parameters, it is dramatically smaller than modern general-purpose language models and should not be expected to match their capabilities.
It may:
- Generate incoherent text
- Repeat phrases
- Produce factual errors
- Struggle with complex instructions
- Have limited world knowledge
- Perform poorly on difficult reasoning tasks
- Produce grammatically unusual text
- Fail to maintain coherent long-form generations
A 2,048-token context window does not eliminate the limitations caused by the model's small parameter count.
This model should be considered a research and experimentation model, rather than a production-ready general-purpose LLM.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Novi-AI/Novi-Micro-Base"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
)
prompt = "Hello, my name is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=50,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Because Novi-Micro uses a custom architecture, trust_remote_code=True is required when loading the model through Transformers.
Project History
Novi-Micro-Base is part of Project Kairo, the development codename for the Novi model project.
The Novi series follows the earlier AppleMind experiments and represents the primary model-development line of Novi-AI.
AppleMind → Novi-Nano → Novi-Micro → future Novi models 🚀
Novi-Micro substantially expands upon Novi-Nano by increasing the model to approximately 5.04M parameters, expanding the context window from 256 to 2,048 tokens, and training on approximately 1 billion tokens.
Training Infrastructure
The model was trained using GPU compute through the BananaAll training environment.
The training worker supports GPU acceleration through CUDA and Intel XPU, with Torch compilation enabled for supported environments.
This model was trained from scratch rather than fine-tuned from an existing language model checkpoint.
Acknowledgements
Novi-Micro was built using the open-source machine-learning ecosystem and datasets made available by the community.
Special thanks to:
- Hugging Face 🤗
- FineWeb
- FineWeb-HQ
- The open-source Transformers ecosystem
License
This model is released under the Apache 2.0 license.
🧠 Novi AI
Small models. Big experiments.
Novi-Micro is intentionally tiny — exploring how far a language model can go with only a few million parameters and approximately one billion training tokens.
Novi AI 2026 — Project Kairo