MTLM2-40M (TinyGPT)
MTLM2-40M is a highly experimental, tiny language model (~40 Million parameters) designed as a research artifact to explore the lower bounds of language modeling capabilities.
The primary research question driving this model was:
Can a microscopic model (40M params), when saturated with a large amount of data (14B tokens), generate minimally coherent narrative text?
The answer is yes. Validated by a Perplexity of 54.21 on WikiText-2, the model demonstrates surprising structural and narrative cohesion for its size.
📊 Model Details
- Architecture: TinyGPT (Custom
Llama-style architecture with tweaks for small-scale efficiency). - Parameters: ~41.5 Million.
- Training Data: 14 Billion Tokens (English).
- Training Framework: JAX / Flax (XLA).
- Hardware: Google Cloud TPU v5e-8.
- Training Time: ~6 Hours.
- Precision: FP32 (Exported to Safetensors).
📉 Benchmarks
| Benchmark | Metric | Result |
|---|---|---|
| WikiText-2 | Perplexity (PPL) | 54.21 |
Note: Evaluation performed using sliding window approach. The low PPL confirms strong grammatical alignment despite the small parameter count.
🧪 Capabilities & Limitations
What it CAN do:
- Narrative Cohesion: It can generate grammatically correct English sentences that flow logically.
- High Saturation: Trained on ~350 tokens per parameter (far exceeding Chinchilla scaling laws), making it extremely "dense".
What it CANNOT do:
- Reasoning/Logic: It cannot solve math problems, code, or answer complex logical queries.
- Instruction Following: It has NOT undergone Supervised Fine-Tuning (SFT). It is a base completion model.
Usage
This model requires trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Madras1/MTLM2-40M"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0]))
Author Developed by Madras1 (Gabriel).