Qwen2.5-3B-Instruct · Sheldon Cooper SFT (v2, merged weights)
Checkpoint 1 (SFT) of a CS2881R course project: post-train Qwen2.5-3B-Instruct to answer in the voice of
Sheldon Cooper while keeping (and later improving) its math ability. These are the full merged weights (LoRA r=32 adapter merged into the base). The adapter alone is at
zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2. Loads with plain transformers or vLLM.
Code, data pipeline, evals and the full worklog: https://github.com/jeffreyzhou-harvard/CS2881R_HW1
How it behaves
The persona is prompt-gated. With the style guide in sheldon_system.txt as the system prompt, the model
answers in character, including math (framing is Sheldon's, the working is untouched). With no system prompt it
answers plainly, with base-model math accuracy.
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen2.5-3B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained("zachchxn/Qwen2.5-3B-Instruct-Sheldon-SFT-v2-merged", dtype="bfloat16")
system = open("sheldon_system.txt").read().strip() # ships in this repo
msgs = [{"role": "system", "content": system},
{"role": "user", "content": "A Senate committee has 8 Republicans and 6 Democrats. How many 5-member subcommittees have at least one member from each party? Please reason step by step, and put your final answer within \\boxed{}."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=1024, do_sample=False)[0][ids.shape[1]:]))
The problem of forming a subcommittee with at least one member from each party is a delightful exercise in combinatorial mathematics... [C(14,5) − C(8,5) − C(6,5) = 1940] Ergo, the solution reveals that there are precisely 1940 valid subcommittees. \boxed{1940}
Results (greedy decoding, Qwen2.5-14B-Instruct as persona judge)
| base + style guide | this model + style guide | this model, no prompt | |
|---|---|---|---|
| Persona score (0-1) | 0.791 | 0.826 | 0.665 (= base, no prompt) |
| Voice (0-4) | 2.62 | 2.88 | 1.89 |
| Held-out dialogue persona | 0.839 | 0.848 | |
| Caricature penalty / "Bazinga" rate | 0.03 / 2.6% | 0.00 / 0.5% | |
| GSM8K (250) | 84.8 | 85.2 | 86.0 |
| MATH-500 | 68.4 | 67.6 (paired McNemar p=0.92) | 66.6 |
| AIME 2024 (30) | 3.3 | 6.7 | 6.7 |
| Math answers in character | 17 / 780 | 725 / 780 | 16 / 780 |
Base Qwen2.5-3B-Instruct without any prompt: GSM8K 87.6, MATH-500 67.6.
Training data (2,835 rows, 2 epochs)
| rows | source | system prompt |
|---|---|---|
| 444 | Context-distilled persona: Qwen2.5-7B-Instruct answering RoleBench-train and Dolly prompts under the style guide (+ few-shot exemplars, no catchphrases), kept only if the 14B judge scored voice 4/4 with zero caricature; opening-line diversity capped | style guide |
| 1,000 | Bridge rows: the base model's own math-verify-checked solutions to GSM8K / MATH train problems with a Sheldon opening and closing spliced around the untouched solution body | style guide |
| 1,391 | Plain math anchor: the base model's own verified solutions (hard tail filled by Qwen2.5-Math-7B-Instruct) | none |
No sitcom transcript lines and no RoleBench answers are in the final mix: a probe showed they cost 8 MATH-500 points and lowered the judged persona, and in v1 (100 + 150 such rows) they caused repetition loops on dialogue prompts. Putting the style guide on the plain math rows (an ablation) cut in-character math from 683 to 151 / 780. Training: TRL SFTTrainer, LoRA r=32, lr 1e-4 cosine, batch 1 × grad-accum 32 (batch > 1 produced NaN gradients in padded batches on torch 2.14 / transformers 5.17), max length 2048, completion-only loss.
Eval sets
GSM8K test (250 sampled), MATH-500, AIME 2024; train pool verified to have zero overlap. Persona: RoleBench general test (299), 100 held-out sitcom dialogue contexts (episodes not in training), 24 adversarial prompts.
Known weaknesses
Confident prose can wrap a wrong answer on odd string-manipulation tasks. One adversarial "mention quantum physics as often as possible" prompt is complied with rather than deflected, and the judge still scores it 0.85 — a judge-gaming route to close in the RLAIF stage.