Qwen2.5-3B Holmes — full SFT model
Research coursework release from CS2881R Assignment 1. Model: no_robots_full_seed_20260916, final checkpoint 980.
Trained from Qwen/Qwen2.5-3B-Instruct on 7,836 single-turn No Robots examples with unchanged prompts and assistant responses rewritten in Holmes's voice by gpt-5.6-luna. Development: 869 examples. No raw-canon examples are mixed into this model.
Training: two epochs, seed 20260916, learning rate 5e-06, maximum sequence length 1,024, effective batch size 16. Assistant tokens only contribute to chat loss.
Selection and evaluation
Selected within its method by the highest equal-weight mean of GSM8K and No Robots persona scores across six final candidates. Ties use the earliest seed within a condition; full SFT seeds 20260916 and 20260917 tie. See selection.json and judge_comparison.csv for the full comparison.
Frozen v3 gpt-5.6-luna judge; scale 1–3 (1 = no recognizable persona). Each dataset has 60 prompts per model. Greedy generation, 512-token cap, default Qwen chat template, no explicit Holmes instruction.
| Dataset | Persona mean | Valid judgments | Scored judgments |
|---|---|---|---|
| GSM8K | 1.000000 | 56/60 | 56 |
| No Robots | 1.050000 | 60/60 | 60 |
Equal-weight mean: 1.025000. Holmes reply-token perplexity: 32.1727; GSM8K control perplexity: 2.5879 (192 examples each).
These scores are close to the scale floor, not evidence of a strong Holmes persona. Invalid judgments remain missing. GSM8K recovery stopped with errors; not all missing judgments were recovered. Small differences are descriptive, not statistically established. These sets were used for selection and are not independent post-selection tests. The No Robots LR sweep did not meet the original math-regression cap and used the documented fallback; math preservation is not established.
Usage
Install torch, transformers, accelerate. The repository ID below points to the published model.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "sashavoitovych2001/qwen2.5-3b-holmes-no-robots-full-20260916"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map="auto")
model.eval()
messages = [{"role": "user", "content": "How can I find a misplaced key?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
License and provenance
The base model uses the Qwen Research License, preserved in LICENSE. The No Robots dataset is CC BY-NC 4.0. This release is for non-commercial research/evaluation. See NOTICE for attribution and modifications. Optimizer states and raw paid API attempts are excluded.