230912GPT2_fine_tuned_SP_GPT2_config_ESM_tokenizer_6kSwissProt_20epochs

This model is a fine-tuned version of gpt2 on a curated set of ∼6,000 SwissProt signal peptides, which include all mammalian signal peptides that have been experimentally verified and deposited on SwissProt. It achieves the following results (negative log likelihood on next-token prediction) on the evaluation set:

• Loss: 2.1215

Model description

Base model: https://huggingface.co/docs/transformers/en/model_doc/gpt2.

Intended uses & limitations

Generation of novel mammalian signal peptides to fine-tune the secretion of specific recombinant proteins within a host cell.

Training and evaluation data

a curated set of ∼6,000 SwissProt signal peptides, which include all mammalian signal peptides that have been experimentally verified and deposited on SwissProt. The data was downloaded from the public UniProt database with accession date 12/09/2023.

Training procedure

Causal language modelling (CLM) recipe from HuggingFace's Trainer. Train test split 80/20.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 531
• num_epochs: 20

Training results

Framework versions

• Transformers 4.33.1
• Pytorch 2.0.1+cu118
• Datasets 2.14.5
• Tokenizers 0.13.3