Model Card for Puffin-Phi V2
This is my first fine-tune of Puffin that seems to be working fairly reliably!
Model Details
Model Sources
This model was trained on the Puffin Dataset, made by LDJ, using a slightly modified version of the dataset that removed >2000 token entries, so there would be no early cutoffs during training phi, since it's context is 2k.
Uses
Let me know!
How to Get Started with the Model
Phi does not support device_map "auto", and does not seem to want to inference in fp16, so use bf16.
Here is working code to inference, though it can be improved:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
sysprompt = "The assistant gives helpful, detailed, and polite answers to the user's questions.\n"
model = AutoModelForCausalLM.from_pretrained("teknium/Puffin-Phi-v2", trust_remote_code=True, torch_dtype=torch.bfloat16).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("teknium/Puffin-Phi-v2", trust_remote_code=True, torch_dtype=torch.bfloat16)
inputs = tokenizer(f"{sysprompt}USER: Write a negative review for the website Twitter.\nASSISTANT:", return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs, max_length=128, do_sample=True, temperature=0.2, top_p=0.9, use_cache=True, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]
print(text)
The prompt format is ShareGPT/Vicuna, so it uses the sysprompt (defualt in sysprompt variable) then is prompted like so:
USER: <prompt>
ASSISTANT:
Training Details
Training Procedure
Trained with Axolotl. View the wandb runs for all my puffin runs (this is puffin-phi-4 on wandb): https://wandb.ai/teknium1/puffin-phi/runs/puffin-phi-4
Evaluation
TODO