Vertex 0.1 (10M)
Small LLaMA-architecture causal language model, ~10.9M parameters.
Model Details
- Architecture: LlamaForCausalLM
- Parameters: ~10.9M
- Hidden size: 256
- Layers: 4
- Attention heads: 4 (2 KV heads, GQA)
- Head dim: 64
- Intermediate size: 640
- Max position embeddings: 1024
- Vocab size: 32000
- Dtype: float32
Training
- Epochs: 3
- Global steps: 3207
- Batch size: 8
- Estimated tokens seen: ~26M total, ~8.8M tokens/epoch
- Training time: ~1 Hour
- Hardware: M5 Pro 24gb Macbook
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.1-10M-base")
tokenizer = AutoTokenizer.from_pretrained("VertexResearch/Vertex-0.1-10M-base")
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
This is a very small model (~10.9M params) trained on a small dataset. Expect limited coherence, knowledge, and generalization compared to larger models. Intended for experimentation, education, and low-resource testing, not production use.