GPT-2 XL Alpaca, one-epoch checkpoint
This is an experimental instruction-tuned GPT-2 XL checkpoint. It starts from the pretrained openai-community/gpt2-xl weights and has completed one epoch of supervised fine-tuning on the training split of yahma/alpaca-cleaned. I’m publishing this checkpoint to preserve the experiment and evaluate its behavior.
Training
- Base model: GPT-2 XL, approximately 1.5 billion parameters
- Instruction data:
yahma/alpaca-cleaned - Split: 95% training, 5% validation
- Completed training: one epoch
- Sequence length: 512 tokens
- Objective: predict response tokens given the instruction and optional input; prompt and padding tokens are excluded from the loss
This card describes the uploaded checkpoint only. Continued pretraining on Dolma is a separate experiment and is not claimed for these weights.
Prompt format
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Explain what a transistor does.
### Response:
For a task with additional context, insert an ### Input: section between the instruction and response sections.
Use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "YOUR_USERNAME/YOUR_REPO"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
model.eval()
prompt = (
"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\nExplain what a transistor does.\n\n"
"### Response:\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=120,
do_sample=True,
temperature=0.7,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Evaluation and limitations
This is an intermediate research checkpoint. Its instruction following, factual accuracy, and general usefulness have not yet been systematically evaluated. GPT-2 XL has a limited context window, and the training examples were capped at 512 tokens. Generated answers may be incorrect, incomplete, repetitive, or biased. Review outputs before relying on them.