bert-tiny (mirror)
A 2-layer, 128-hidden BERT — about 4.4M parameters / 17 MB. Small enough to fine-tune on a laptop CPU in minutes, which makes it the go-to model for smoke tests, CI pipelines, unit tests for training code, and edge deployment.
[!NOTE] This is a mirror. The weights and tokenizer files here are an unmodified copy of
prajjwal1/bert-tiny, re-hosted on this profile for reproducibility and convenience. All credit for the original work belongs to its authors. The upstream license (mit) is preserved and applies to this copy. If you need the canonical version, please use the upstream repository.
Specs
| Layers | 2 |
| Hidden size | 128 |
| Attention heads | 2 |
| Parameters | ~4.4M |
| Vocab | 30,522 (uncased WordPiece) |
| Disk | ~17 MB |
Usage
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("priyaganesh2050/bert-tiny")
model = AutoModel.from_pretrained("priyaganesh2050/bert-tiny")
out = model(**tok("A tiny BERT for fast experiments.", return_tensors="pt"))
print(out.last_hidden_state.shape) # torch.Size([1, 9, 128])
Fine-tuning for classification:
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("priyaganesh2050/bert-tiny", num_labels=2)
When to use this
- Good for: CI/CD tests of training loops, hyperparameter search, teaching, edge/mobile, latency-critical baselines.
- Not good for: accuracy-sensitive production NLP. A 2-layer model gives up a lot of quality
versus
bert-base. Use it as a baseline, then scale up.
Citation
The tiny BERT variants come from the well-read-students line of work:
@misc{turc2019,
title = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
year = {2019},
eprint = {1908.08962},
archivePrefix = {arXiv}
}