Nano-Llama
A compact 67M parameter LLaMA-2-style language model pretrained on FineWeb dataset.
Model Details
- Architecture: LLaMA-2-style transformer
- Parameters: 678M
- Training Data: FineWeb dataset (~100M tokens)
- Context Length: 1024 tokens
- Layers: 6
- Hidden Size: 768
- Attention Heads: 12
Training
- Dataset: FineWeb (web-crawled high-quality text)
- Tokens Trained: ~110M tokens
- Training Time: ~6 hours on RTX 3090
- Optimizer: AdamW
- Learning Rate: 1e-4
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("vishesh-t27/Nano-Llama")
model = AutoModelForCausalLM.from_pretrained("vishesh-t27/Nano-Llama")
model.eval()
# Test prompt
text = "The future of artificial intelligence is"
inputs = tokenizer(text, return_tensors="pt")
# Generate text
outputs = model.generate(
**inputs,
max_new_tokens=50,
temperature=0.8,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
# Decode and print
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
Limitations
- Small model size (67M parameters)
- Limited training data compared to larger models
- May generate repetitive or nonsensical text
License
MIT License