Stalk-Mini
A 43.66M parameter Llama-architecture causal language model trained from scratch on TinyStories.
Model Details
Model Description
Stalk-Mini is a compact causal language model built on the Llama architecture. Trained from scratch as an experimental model for low-latency inference on custom architectures.
- Developed by: Ventie
- Model type: LlamaForCausalLM
- Language(s) (NLP): English
- License: MIT
- Architecture: 43.66M Parameters
- Vocabulary Size: 16,000 (Custom BPE Tokenizer)
Uses
Direct Use
Intended for lightweight text generation benchmarks, local inference speed tests, and experimental LLM architecture evaluation.
Out-of-Scope Use
Not suitable for complex reasoning, instruction following, or production tasks requiring factual accuracy.
Training Details
Training Data
Trained on 50,000 samples from the TinyStories dataset.
Training Procedure
- Epochs: 2.0
- Batch Size / Steps: 3,126 steps
- Final Training Loss: ~1.54
- Optimizer: AdamW with cosine schedule
Technical Specifications
Compute Infrastructure
- Software Stack: PyTorch (CUDA 13.0), Hugging Face Transformers