VinaySLM (tiny)
VinaySLM is a small language model (SLM) built from scratch in PyTorch to learn how decoder-only Transformers work end to end -- tokenization, embeddings, causal self-attention, training, and generation -- without importing a pretrained architecture. This is not a production-quality language model.
What this is
- A tiny, hand-written, decoder-only (GPT-style) Transformer.
- 466,176 parameters.
- Trained on a ~600-token toy corpus of AI/programming sentences.
- Packaged here as a custom Hugging Face model (
trust_remote_code=True) so it can be loaded and inspected through thetransformersecosystem, while the actual Transformer implementation (modeling_vinayslm.py) is a thin wrapper around this project's own, unmodified model code.
Architecture
| Type | Decoder-only Transformer (GPT-style, pre-LayerNorm) |
| Parameters | 466,176 |
| Vocabulary size | 255 |
| Context length | 32 |
| Embedding dimension | 128 |
| Transformer layers | 2 |
| Attention heads | 4 |
| Feed-forward dimension | 512 |
| Dropout | 0.0 |
| Positional encoding | Learned absolute positional embeddings |
| LM head | Untied from the token embedding |
| Decoding | Greedy (argmax) only -- no sampling yet |
Tokenizer
A minimal, from-scratch word-level tokenizer (VinaySLMTokenizer,
wrapping this project's WordTokenizer): text is lowercased and split into
words and punctuation with a regex; the vocabulary was built directly from
the training corpus. <pad>, <unk>, <bos>, <eos> are the only
special tokens. This is not a BPE or pretrained tokenizer.
Training data
A tiny, hand-written corpus of AI/programming sentences (see this
project's data/raw/toy_corpus.txt) -- a few hundred tokens total.
Important limitation
The model heavily overfits this tiny toy corpus. It was trained to memorize a few dozen short sentences, not to generalize. Prompts close to a training sentence will often reproduce that sentence verbatim; novel prompts will produce low-quality or repetitive continuations. This model exists to demonstrate a correct, from-scratch Transformer pipeline, not to produce useful or general text.
Example
Prompt: "Machine learning"
Output: "machine learning allows computers to learn patterns from data."
How to load
This repository ships its own model/config/tokenizer code (see
configuration_vinayslm.py, modeling_vinayslm.py,
tokenization_vinayslm.py), so loading it requires trust_remote_code=True
and the vinayslm Python package installed (the custom code imports
this project's own TinyGPT/WordTokenizer classes rather than
duplicating them -- see "Limitations" in this project's README):
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)
Or, equivalently, import the classes directly:
from vinayslm.hf import VinaySLMForCausalLM, VinaySLMTokenizer, generate_greedy
model = VinaySLMForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny")
tokenizer = VinaySLMTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny")
print(generate_greedy(model, tokenizer, "Machine learning", max_new_tokens=15))
How to reproduce this export
From the VinaySLM project root, with its .venv active:
python scripts/prepare_data.py
python scripts/train.py
python scripts/export_hf.py
scripts/export_hf.py loads the trained artifact from checkpoints/tiny/,
wraps it in the Hugging Face-compatible classes, verifies it produces
identical greedy output to the original local model, and writes this
directory. Nothing is uploaded to the Hugging Face Hub by this script.