Qwen2.5-3B-Instruct · Sheldon Cooper persona SFT (v3b: chat + existing + generated step-by-step Sheldon math, merged)

Qwen2.5-3B-Instruct fine-tuned with LoRA to answer every request in the voice of Dr. Sheldon Cooper while still completing the task, no system prompt needed. This is the merged bf16 model; the LoRA adapter and all 20 trajectory checkpoints are in agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3b-LoRA.

Code: training, evaluation, and data-generation scripts for all arms are in agastyasridharan/cs2881r-sheldon-sft.

Built for Harvard CS 2881R (persona + verifiable STEM capability, SFT → RLAIF → RLVR). v3b is the third of three SFT arms:

Base model GSM8K: 86.7% (316 tokens per answer). v3b answers math the way the generated data does: a one-sentence pedantic preamble, one arithmetic operation per line, one quip, then \boxed{} on the last line. Example (GSM8K test, correct):

Percentages are just fractions with a different name; I find the distinction more interesting than the arithmetic itself. 80 / 2 = 40 students per class 40 x 0.6 = 24 boys per class Twenty-four boys is a perfectly respectable number, though I'd still rather count them by hand. \boxed{24}

It fixes the format problems of v2/v3a (100% boxed, explicit steps, persona leakage in math down from 98% to 10%) but not the accuracy plateau: v3a, an interim run with half the generated rows (2,320 → 64.7%), and v3b all land at 63–67%, within the ±1-point re-run noise of each other. The remaining gap to the base model is a reasoning regression shared by every SFT arm (only 14% of wrong answers contain an arithmetic slip; the rest misread the problem), which is left to the RL stages.

Training data

Chat (11,910 rows) and existing Sheldon math (2,036 rows) are identical to v3a (see that card). The new component:

Generated step-by-step Sheldon math (4,485 train / 300 val rows). 4,800 problems were sampled from the GSM8K train split (seed 20260909; the test split was never touched, and a number-multiset + word-Jaccard check found no near-duplicates of test problems). Each problem's reference solution was rewritten into Sheldon's voice by Claude Sonnet, one subagent per 25-problem batch, under a strict template: ≤ 2 sentences (≤ 60 words) of preamble; one arithmetic operation per line with a 2–6 word label; ≤ 1-sentence quip; \boxed{N} alone on the last line; 25–320 words. Every row had to pass a deterministic verifier before acceptance:

• boxed answer equals the GSM8K reference; every intermediate value from the reference's <<a op b = c>> annotations appears in the text;
• every pure-arithmetic = line evaluates correctly (fractions and thousands separators handled);
• ≥ 1 Sheldon marker, no chatbot register ("Great question", markdown, LaTeX, bullets), "Bazinga" in ≤ 1 row per batch, unique 2-word openers within a batch;
• for paraphrased prompts: the digit multiset and number-word multiset of the paraphrase equal the original's.

Agents fixed failing rows and re-verified (≤ 3 runs); final acceptance was 4,800 / 4,800. 15 rows were then dropped because the agents flagged the GSM8K reference solution itself as wrong (e.g. 192 × 100 = 1920, steps summing to 108 with answer key 106). Prompt variants: 1,915 verbatim GSM8K questions, 1,531 questions plus the exact eval instruction "Please reason step by step, and put your final answer within \boxed{}.", 1,339 casual-voice paraphrases (typos, backstory, same numbers). Generated-row statistics: mean 71 words, 3.4 equation lines, Big Bang Theory character named in 29.6% of rows, "Bazinga" in 4.0%, most frequent 2-word opener 105 / 4,785 rows.

Final train set: 18,431 rows (35.4% math), 650 val (300 of them generated math, so val loss is not comparable with v2/v3a). 20% of rows carry a generic system prompt, 80% none.

Training recipe (identical to v2/v3a except for the data)

Evaluation (trajectory, zero-shot, greedy)

GSM8K test (1,319 problems), prompt "<question>\nPlease reason step by step, and put your final answer within \boxed{}.", strict = a boxed number is present and equals the reference. Persona metrics are rule-based marker rates (share of responses containing a Big Bang Theory reference: Leonard / Penny / Amy / Howard / Raj / Meemaw / Bazinga / roommate agreement / …) on the 502 held-out prompts from the training distribution and 40 short out-of-distribution prompts. Re-running a checkpoint moves GSM8K by about ±1 point (batched greedy decoding).

* val loss of the base model on this run's validation split (includes 300 generated math rows).

Reading the curve. Step 29 is the same transient collapse seen in every arm (35.1%, with 23% of answers running to the 1,024-token cap) and is gone by step 58. From step ~200 the model sits at 62–65% GSM8K with 100% boxed answers, 3.5 equation lines per answer and 23 words before the first digit (v3a: 0.7 lines, 80 words). Doubling the generated data from 2,320 rows (interim run, 64.7%) to 4,485 rows changed nothing on accuracy. Persona strength on chat prompts matches v2/v3a (~60% held-out marker rate); persona leakage into math answers is ~10× lower than v3a. The accuracy drop versus the base model is uniform across problem difficulty (2-step problems 93 → 77%, 4-step 83 → 60%, 6-step 74 → 33%), and the three v3 arms together solve 1,117 of the 1,144 problems the base model solves, so the capability is retained and the fine-tuned sampler is simply noisier.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
m = "agastyasridharan/Qwen2.5-3B-Instruct-Sheldon-SFT-v3b"
tok = AutoTokenizer.from_pretrained(m); model = AutoModelForCausalLM.from_pretrained(m, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "A baker makes 24 muffins and sells 3/4 of them. How many are left?\nPlease reason step by step, and put your final answer within \\boxed{}."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Greedy decoding is fine for math (the format is stable). For chat, sampling at T≈0.7 diversifies openers ("Excuse me, but …" dominates under greedy). The model invents trivia in character; do not rely on its factual asides.

Known limitations

• 3B model; GSM8K is ~23 points below the base model. This is the pre-RL SFT stage; the generated data fixed format, not reasoning.
• Math answers are deliberately terse (one line per operation); the model does not show verbal reasoning between equations.
• Inherits the Qwen Research License from the base model.