GPT-3 Small (125M) - Paper-Faithful Recreation

Welcome! This repository hosts a 125-million parameter language model, an attempt to faithfully recreate the architecture from the original OpenAI GPT-3 paper ("Language Models are Few-Shot Learners").

The model was trained on the FineWeb dataset using PyTorch and the Hugging Face Transformers library. This repository contains the raw model weights (.pt file) and the exact code required for inference.

Important Architectural Note

This model uses a specific, paper-faithful architecture that differs from the standard Hugging Face GPT2Model AND my other models. Notably, it uses Conv1D layers for attention projections, as described in the original paper.

Due to subtle incompatibilities between this custom architecture and the high-level AutoModelForCausalLM.from_pretrained() function, loading this model requires using the provided inference.py script, which defines the architecture and loads the weights directly. Standard Hugging Face loading methods will not work.

You can find all code on GitHub

This is the 125-million parameter model (an attempt to replicate GPT-3 Small).

Model Details

• Release Date: June 18, 2025
• Parameters: ~125 Million
• Architecture: Paper-faithful GPT-3 Small (using Conv1D for attention projections).
• Training: Trained on a subset of the FineWeb dataset. Further details can be found in the training scripts in the GitHub repository.

This model demonstrates (almost) coherent text generation abilities and serves as a solid foundation for further fine-tuning and experimentation. The model scores 27.98% on MMLU.

How to Use This Model

Because this model requires a specific architecture definition, you must use the provided code to run inference. The standard from_pretrained method will fail.

1. Clone the repository or download the files: You will need inference.py and the model.pt checkpoint file.

2. Run the following Python code: The inference.py script contains all necessary logic. Simply run it from your terminal.

   # Make sure you have torch and transformers installed
   pip install torch transformers
   
   # Run the interactive inference script
   python inference.py
   

   The script will load the model and prompt you for input.

   HOW TO RUN

   git clone https://huggingface.co/k050506koch/GPT3-dev-125m-1806/
   
   #OR
   
   huggingface-cli download k050506koch/GPT3-dev-125m-1806
   cd GPT3-dev-125m-1806/
   git lfs pull
   
   python inference_v2.py
   

   Example Code Structure ( inference_v2.py):

   import torch
   from transformers import GPT2TokenizerFast
   
   # --- 1. The full, correct model architecture is defined here ---
   # (GPT3DevConfig, GPT3DevAttention, GPT3DevLM, etc.)
   
   def make_model(vocab_size):
       # ... function to create model instance ...
   
   # --- 2. The model is loaded directly from the .pt file ---
   CHECKPOINT_PATH = './emergency_ckpt_jetson/model.pt'
   tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
   model = make_model(tokenizer.vocab_size)
   model.load_state_dict(torch.load(CHECKPOINT_PATH))
   model.eval()
   
   # --- 3. The script runs a generation loop ---
   # (The code then prompts for input and generates text)
   

Contributing

Contributions are welcome! I'm a student interested in AI, so my code may have both syntax and logic issues. Please open an issue or submit a pull request for any improvements or bug fixes. I'd be happy to see them.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgements

A huge thank you to the teams and communities behind these incredible projects for making this possible!

• OpenAI GPT-3 Paper
• Hugging Face Transformers
• PyTorch