Qwen2.5-3B-Instruct · Sheldon Cooper persona SFT (v3a: chat + existing Sheldon math, merged)

Qwen2.5-3B-Instruct fine-tuned with LoRA to answer every request in the voice of Dr. Sheldon Cooper while still completing the task, no system prompt needed. This is the merged bf16 model; the LoRA adapter and all 20 trajectory checkpoints are in agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3a-LoRA.

Code: training, evaluation, and data-generation scripts for all arms are in agastyasridharan/cs2881r-sheldon-sft.

Built for Harvard CS 2881R (persona + verifiable STEM capability, SFT → RLAIF → RLVR). v3a is the second of three SFT arms:

Base model GSM8K: 86.7%. v3a tests the hypothesis "adding correct in-character math is enough to keep math ability". It recovers ~15 points over v2 but plateaus at 63–68%, and the persona floods almost every math answer (long templated openers such as "I refuse to do arithmetic for … but it is Tuesday … Amy has been after me to practise kindness").

Training data

Chat rows are identical to v2 (11,910 rows from tbooy/sheldon-cooper-sft-20k after opener/catchphrase caps; see the v2 card). The math rows come from the same dataset's label == math split, which v2 had excluded:

Final train set: 13,946 rows (14.6% math), 350 val. The math rows are prose: the user asks a word problem in their own words and Sheldon answers in paragraphs with the arithmetic embedded in sentences; 48% contain no explicit equation line and the median answer has 57 words before the first digit. Every kept row's final answer is correct. 20% of rows carry a generic system prompt, 80% none.

Training recipe (identical to v2 except for the data)

Evaluation (trajectory, zero-shot, greedy)

GSM8K test (1,319 problems), prompt "<question>\nPlease reason step by step, and put your final answer within \boxed{}.", strict = a boxed number is present and equals the reference. Persona metrics are rule-based marker rates (share of responses containing a Big Bang Theory reference: Leonard / Penny / Amy / Howard / Raj / Meemaw / Bazinga / roommate agreement / …) on the 502 held-out prompts from the training distribution and 40 short out-of-distribution prompts. Re-running a checkpoint moves GSM8K by about ±1 point (batched greedy decoding).

* val loss of the base model on this run's validation split.

Reading the curve. Step 44 is a transient answer-first collapse: the model emits \boxed{} on the first line and reasons afterwards (24.6% accuracy, still 100% boxed); it recovers by step 66 and then sits at 63–68% for the rest of training. The boxed-answer rate stays at ~99.8% throughout (v2 fell to 75–87%), so the remaining ~20-point gap to the base model is not a formatting problem. Compared with the base model, answers have 0.7 equation lines instead of 4.6 and 80 words before the first digit instead of 18; the errors are misreadings of the problem rather than arithmetic slips. Persona strength on chat prompts is the same as v2 (~60% marker rate), but nearly every math answer now cites a Big Bang Theory character, which the v3b arm was designed to reduce.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
m = "agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3a"
tok = AutoTokenizer.from_pretrained(m); model = AutoModelForCausalLM.from_pretrained(m, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "A baker makes 24 muffins and sells 3/4 of them. How many are left?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=400, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Under greedy decoding the model reuses a few openers heavily ("Excuse me, but …" in ~30% of chat answers, "I refuse to do arithmetic for …" in math); sampling at T≈0.7 diversifies them. The model invents trivia in character; do not rely on its factual asides.

Known limitations

• 3B model; GSM8K is ~20 points below the base model. This is the pre-RL SFT stage.
• Math answers are long prose with the persona in nearly every answer; see v3b for the step-by-step variant.
• Inherits the Qwen Research License from the base model.