Advertisement-Description-Generator gpt2 finetuned on product descriptions
Model Details
This model generates a 1-2 line description of a product given the keywords associated with it. It is a fine-tuned version of GPT-2 model on a custom dataset consisting of
328 rows in the format: (keyword, text)
Example:
[Echo Dot,smart speaker,Alexa,improved sound,sleek design,home], Introducing the all-new Echo Dot - the smart speaker with Alexa. It has improved sound and a sleek design that fits anywhere in your home.
- Developed by: Nihal Srivastava
- Model type: Finetuned GPT-2
- License: MIT
- Finetuned from model [optional]: GPT-2
Uses
Inital Imports of the model and setting up helper functions:
from transformers import AutoTokenizer, AutoModelForCausalLM
def join_keywords(keywords, randomize=True):
N = len(keywords)
if randomize:
M = random.choice(range(N+1))
keywords = keywords[:M]
random.shuffle(keywords)
return ','.join(keywords)
SPECIAL_TOKENS = { "bos_token": "<|BOS|>",
"eos_token": "<|EOS|>",
"unk_token": "<|UNK|>",
"pad_token": "<|PAD|>",
"sep_token": "<|SEP|>"}
device = torch.device("cuda")
tokenizer = AutoTokenizer.from_pretrained("NihalSrivastava/advertisement-description-generator")
model = AutoModelForCausalLM.from_pretrained("NihalSrivastava/advertisement-description-generator").to(device)
keywords = ['laptop', 'fast', 'gaming', 'great graphics', 'affordable']
kw = join_keywords(keywords, randomize=False)
prompt = SPECIAL_TOKENS['bos_token'] + kw + SPECIAL_TOKENS['sep_token']
generated = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)
device = torch.device("cuda")
generated = generated.to(device)
model.eval();
To generate a single best description:
# Beam-search text generation:
sample_outputs = model.generate(generated,
do_sample=True,
max_length=768,
num_beams=5,
repetition_penalty=5.0,
early_stopping=True,
num_return_sequences=1
)
for i, sample_output in enumerate(sample_outputs):
text = tokenizer.decode(sample_output, skip_special_tokens=True)
a = len(','.join(keywords))
print("{}: {}\n\n".format(i+1, text[a:]))
Output:
1: The gaming laptop has a fast clock speed, high refresh rate display, and advanced connectivity options for unparalleled gaming performance.
To generate top 10 best descriptions:
# Top-p (nucleus) text generation (10 samples):
sample_outputs = model.generate(generated,
do_sample=True,
min_length=50,
max_length=MAXLEN,
top_k=30,
top_p=0.7,
temperature=0.9,
repetition_penalty=2.0,
num_return_sequences=10
)
for i, sample_output in enumerate(sample_outputs):
text = tokenizer.decode(sample_output, skip_special_tokens=True)
a = len(','.join(keywords))
print("{}: {}\n\n".format(i+1, text[a:]))
Output: Will generate 10 best outouts of product description as above.
Training Hyperparameters
DEBUG = False
USE_APEX = True
APEX_OPT_LEVEL = 'O1'
MODEL = 'gpt2' #{gpt2, gpt2-medium, gpt2-large, gpt2-xl}
UNFREEZE_LAST_N = 6 #The last N layers to unfreeze for training
SPECIAL_TOKENS = { "bos_token": "<|BOS|>",
"eos_token": "<|EOS|>",
"unk_token": "<|UNK|>",
"pad_token": "<|PAD|>",
"sep_token": "<|SEP|>"}
MAXLEN = 768 #{768, 1024, 1280, 1600}
TRAIN_SIZE = 0.8
if USE_APEX:
TRAIN_BATCHSIZE = 4
BATCH_UPDATE = 16
else:
TRAIN_BATCHSIZE = 2
BATCH_UPDATE = 32
EPOCHS = 20
LR = 5e-4
EPS = 1e-8
WARMUP_STEPS = 1e2
SEED = 2020
EVALUATION_STRATEGY=epoch
SAVE_STRATEGY=epoch
Model Card Contact
Email: nihal.srivastava05@gmail.com Github: https://github.com/Nihal-Srivastava05 LinkedIn: https://www.linkedin.com/in/nihal-srivastava-7708a71b7/