RegexGym-Qwen3-4B
Qwen3-4B fine-tuned to write regexes from examples. Give it strings that should match and strings that shouldn't; it returns one regex that separates them. It was trained on regexgym-verified-traces — teacher reasoning traces where every kept example actually solved a hidden holdout, checked by running the regex, not by a learned reward.
How it does
120 held-out tasks, decontaminated, strict pass@1:
| System | pass@1 | mean hidden acc |
|---|---|---|
always answer .* | 0% | 0.50 |
| classical regex induction | 10% | 0.59 |
| base Qwen3-4B | 37% | 0.63 |
| this model | 44% | 0.82 |
| Gemma-4-31B (teacher, 8× bigger) | 51% | 0.70 |
What the numbers mean. pass@1 is all-or-nothing per task — the regex has to match every held-out positive and reject every held-out negative to count. It's the strict headline number: 44% means a fully-correct regex on 44 of 100 unseen tasks. mean hidden accuracy is partial credit — on average, the fraction of held-out strings each regex classifies correctly — so a close-but-imperfect answer still scores. That's why it runs higher (0.82) than pass@1: the model is usually close even when it isn't exact.
Seven points over the base it started from. It doesn't reach the teacher — a 4B chasing a 31B has a real gap, and more distillation didn't close it. The writeup covers what I tried (more data, a bigger base, a light GRPO run) and why each one stalled.
Using it
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ctokx/regexgym-qwen3-4b")
model = AutoModelForCausalLM.from_pretrained("ctokx/regexgym-qwen3-4b", torch_dtype="auto",
device_map="auto")
prompt = (
"Produce a single regular expression (in a ```regex block) that matches every POSITIVE "
"and no NEGATIVE example.\n\nPOSITIVE:\n 90210\n 10001-1234\n 33101\n\n"
"NEGATIVE:\n 9021\n ABCDE\n 100011234\n"
)
msgs = [{"role": "user", "content": prompt}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
enable_thinking=False)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
It reasons first, then puts the pattern in a fenced ```regex block.
Training
Full fine-tune of Qwen3-4B, three epochs, on the 660-trace train split. Reasoning targets are capped short so the answer never falls off the end of the sequence (the mistake that made an earlier version score 0%). Eval only ever reports clean pass@1 — never the shaped RL reward.
Run on Modal, one A100 80GB — the fine-tune takes a few minutes. The training traces were generated by Gemma-4-31B on Friendli AI, filtered by an execution verifier before anything reached the training set. Code and pipeline: github.com/ctokx/regexgym.
License
Qwen3-4B is Apache-2.0. The training traces were written by a Gemma teacher, so this model falls under the Gemma Terms of Use. Read them before redistributing.