glubbot1.1-7.5m
Model Description
glubbot1.1-7.5m is an experimental, ultra-lightweight text generation model developed by the Glubbot-Team.
With only 7.5 million parameters, this model is designed for experimental testing, low-resource inference benchmarking, and studying language acquisition in small-scale models. It was trained specifically to generate short, grammatically correct narratives.
- Developed by: Glubbot-Team
- Model Type: Transformer-based Causal Language Model
- Language(s): English
- License: Apache 2.0
- Status: Experimental / Testing
Intended Uses & Limitations
This model is intended solely for research, architectural testing, and experimental evaluation.
Intended Use
- Synthetic text generation for short, child-like stories.
- Benchmarking inference speeds on edge devices or CPUs.
- Testing tokenization strategies and pipeline integration.
Limitations & Biases
- Domain Specificity: Optimized purely for short stories; it cannot handle reasoning, coding, or factual factual queries.
- Limited Capacity: Due to the 7.5M parameter size, the model may produce repetitive structures or drift logically during long generations.
- Not for Production: Do not deploy this model for general-purpose assistant tasks.
Training Details
Training Data
The model was trained on the TinyStories dataset (roneneldan/TinyStories). This dataset consists of short stories containing words typically understood by 3- to 4-year-olds, optimized to teach models basic grammar and reasoning.
Hyperparameters
- Context Length: [e.g., 512 / 1024]
- Batch Size: [Insert Batch Size]
- Learning Rate: [Insert Learning Rate]
- Optimizer: AdamW
Evaluation
Evaluation metrics are monitored primarily to test convergence stability on the TinyStories split. The model is evaluated on its ability to maintain basic grammatical coherence.