Model Card for Model ID
Summary
This is a supervised fine-tuned model for text completion based on Phi 1.5. It has been finetuned on a filtered version of the The Complete Works of William Shakespeare, which can be found and downloaded from here: https://www.gutenberg.org/ebooks/100.
Model Description
- Developed by: Course Organizers
- Finetuned from model: microsoft/phi-1_5
Training Details
This model has been trained using the TLR library and SFTTrainer class from Huggingface.
Training Data
The Complete Works of William Shakespeare, which can be found and downloaded from here: https://www.gutenberg.org/ebooks/100
Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- per_device_train_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 1
- optimizer: Adam with betas=(0.9, 0.95)
- lr_scheduler_type: linear
- weight_decay: 0.1
- num_epochs: 1
Framework Versions
- accelerate==0.26.1
- datasets==2.16.1
- transformers==4.45.2
- trl==0.11.2
Compute Infrastructure and Hardware
Slurm cluster with 8 x H100 Nvidia GPUs.