Model Card for gpjt/8xa100m40-lore-2-both-ordinary-init
This model is gpjt/8xa100m40-lore-2-both-ordinary-init, a trained-from-scratch base model based on
the GPT-2-style architecture from Sebastian Raschka's book
"Build a Large Language Model (from Scratch)",
but extended to optionally use low-rank matrices for the embeddings and the output head --
a technique I've taken to calling LoRE (for Low Rank Embeddings). This idea was
suggested by
AndrewThompson1233.
You can read more about it in the blog post: Low-rank vocab matrices (coming soon!)
LoRE settings for this model
See the blog post for information about what these mean:
- LoRE enabled for token embeddings: True
- LoRE enabled for output head: True
- LoRE rank: 128
- Smart initialization: False
Model Details
Model Description
- Developed by: Giles Thomas, based on code by Sebastian Raschka
- Model type: GPT-2 style Transformer-based causal LLM.
- License: Apache 2
- Parameters: 98,877,184
- Context length: 1,024
- Transformers embedding dimension: 768
- MHA heads: 12
- Layers: 12
- QKV bias: False
- Weight tying: False
Don't have high expectations for the model! It has 163M parameters -- or even fewer if it's using LoRE on either the embeddings or the output head -- and was trained on roughly the Chinchilla-optimal number of tokens (~20x the number of parameters) for a non-LoRE version, which means that it doesn't know many facts and is not terribly smart. If you want to do serious work, use a serious model (I like Qwen's). But if you want to build on this and see what you can do with a 2020-vintage LLM, please do feel free to play with it!
Model Sources
- Repository: gpjt/ddp-base-model-from-scratch
- Blog post: Low-rank vocab matrices (coming soon!)
How to Get Started with the Model
You can download and run the model for inference directly:
from transformers import pipeline
pipe = pipeline("text-generation", model="gpjt/8xa100m40-lore-2-both-ordinary-init", trust_remote_code=True)
out = pipe(
"Every effort moves you",
max_new_tokens=20,
do_sample=True,
temperature=1.4,
top_k=25,
)
print(out[0]["generated_text"])
Note that because it uses custom code, you'll need to set trust_remote_code to True.
It supports AutoTokenizer, AutoModel and AutoModelForCausalLM:
>>> from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("gpjt/8xa100m40-lore-2-both-ordinary-init")
>>> model = AutoModel.from_pretrained("gpjt/8xa100m40-lore-2-both-ordinary-init", trust_remote_code=True)
>>> llm_model = AutoModelForCausalLM.from_pretrained("gpjt/8xa100m40-lore-2-both-ordinary-init", trust_remote_code=True)
You can also fine-tune it; this notebook has an example.
Again, don't expect too much from this model! It's a 163M or fewer-parameter GPT-2 one, trained on a limited number of tokens. It's both dumb and ignorant ;-)
Training Details
- Machine type: 8x A100 on Lambda, with 40 GiB VRAM per GPU.
- Tokens: 3,260,252,160 (the target of 3,260,190,720 -- exactly 20x the non-LoRE model's parameter count -- rounded up to the nearest complete batch).
- Dataset: gpjt/fineweb-gpt2-tokens
- Micro-batch size: 12
- Global batch size: 96
- Dropout: 0.0
- Gradient clipping: 3.5
- Learning rate: 0.0014
- Schedule learning rate: True
- Weight decay: 0.01