Nexara-Nano-100M-Chat
Nexara-Nano-100M-Chat is a tiny Decoder-Only Transformer pretrained and supervised fine-tuned (SFT) from scratch. It is designed solely for experimental and educational purposes.
[!WARNING] This model has only ~110,138,880 parameters and is trained on a small corpus. It is not guaranteed to perform well in production environments. Expect frequent hallucinations and limited factual knowledge.
Model Architecture
- Type: Causal Decoder-Only Transformer (with Rotary Position Embeddings (RoPE), Pre-LN, Tied Embeddings, and GELU activation).
- Parameters: ~110,138,880
- Layers: 12
- Attention Heads: 12
- Embedding Dimension: 768
- Context Length: 512 tokens
- License: MIT License
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("your_username/Nexara-Nano-100M-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("your_username/Nexara-Nano-100M-Chat", trust_remote_code=True)
# Format SFT prompt template
prompt = "### System:\nYou are Nexara, a helpful and polite AI assistant.\n\n### User:\nWhat is your name?\n\n### Assistant:\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))