GPT-2 large, instruction-tuned on SmolTalk
The released GPT-2 large weights (774M parameters) fine-tuned into a chat model on smol-smoltalk, with the loss on assistant replies only. Small, medium and large versions are compared in the instruction-tuning write-up.
Results
| Validation loss (assistant tokens, 1,000 test conversations) | 1.074 |
| Training | one epoch, 460,341 conversations, 14,386 steps of 32 |
| Batch × accumulation, peak learning rate | 16 × 2, 2e-5 |
| Hardware and time | one H100 in bf16, 1.7 hours |
What is the capital of France?
The capital of France is Paris. It is a city known for its rich history and cultural significance. Paris is located in the northern part of France, and it is a major tourist destination.
Files
best.safetensors: the weights at the lowest validation loss (the final step).config.json: the training config.
Loading
These weights use the GPT2 class from gpt2-repro, not Hugging Face
Transformers, and a vocabulary of 50,263 tokens: GPT-2's plus six chat
tokens. Clone gpt2-repro and install its dependencies with
uv, then run the code below from the repository
root:
git clone https://github.com/rahuliyer/gpt2-repro
cd gpt2-repro
uv sync
from huggingface_hub import hf_hub_download
from inference import generate_text
from instruct import get_tokenizer
from instruct.chat import encode_prompt
from instruct.tokenizer import ASSISTANT_END_TOKEN
from model import GPT2, GPT2LargeConfig
tokenizer = get_tokenizer()
path = hf_hub_download("rahuliyer/gpt2-large-smoltalk", "best.safetensors")
model = GPT2.load_from_checkpoint(GPT2LargeConfig(vocab_size=tokenizer.n_vocab), path)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
]
prompt = encode_prompt(tokenizer, messages)
text = generate_text(
model, tokenizer, prompt, max_length=256,
stop_token_id=tokenizer.encode_single_token(ASSISTANT_END_TOKEN),
)
print(text[len(tokenizer.decode(prompt)):])
generate_text samples with temperature 1 and top-5 by default.
Chat format
<|system-start|>...<|system-end|><|user-start|>...<|user-end|><|assistant-start|>
The system message is optional. The model ends its reply with
<|assistant-end|>. Conversations were truncated to 1,024 tokens in training.
Limitations
It is the strongest of the three sizes but still drifts or contradicts itself in longer replies. English only. Not suitable for anything that relies on accurate or safe output.