Qwen3.5-2B · agentic GRPO on BIRD
A 2B text-to-SQL agent, not a single-shot SQL generator. Starting from the
stock Qwen/Qwen3.5-2B (no supervised warm start), it was trained with
GRPO-style RL for 1350 steps against real SQLite execution: it explores a
database with tools, then submits a query, and is rewarded on whether the result
matches the gold answer.
Training data: BIRD train (simple + moderate).
Related models, identical recipe on different training data: cwolff/qwen3.5-2b-grpo-sqale (SQaLe3) and cwolff/qwen3.5-2b-grpo-synsql (SynSQL).
Results — BIRD dev
Measured on this repo as published, loaded straight into vLLM. n=300 BIRD
dev rows (seeded), 6-turn agentic episodes, T=0.6 / top-p 0.95 / top-k 20,
scored by executing against the real SQLite databases.
execution_accuracy_official is BIRD-leaderboard set semantics; strict is
order- and duplicate-sensitive. full puts the DDL in the prompt;
withheld gives the model only the question and makes it discover the
schema through tools.
| stock Qwen3.5-2B | this model | stock Qwen3.8-27B | |
|---|---|---|---|
BIRD full, official | 0.290 | 0.553 | 0.687 |
BIRD full, strict | — | 0.513 | — |
BIRD withheld, official | 0.193 | 0.547 | 0.693 |
BIRD withheld, strict | — | 0.510 | — |
All five bars were produced by the same harness in the same way, which is what makes them comparable. Two things the chart says out loud:
- RL on a 2B works, and works hardest where the schema is hidden. Against its own starting point the gain is +0.263 with the schema in the prompt and +0.354 with it withheld -- exploration is where the training pays.
- A 13x larger untrained model still wins. Stock Qwen3.8-27B reaches 0.687 / 0.693 without any training on this task. If you have the VRAM for a 27B and do not need a 2B's latency or cost, it is the better model; these checkpoints are for the case where you need the small one.
Episode behaviour: submit rate 0.60 / 0.59
(full/withheld), generated SQL executes 97.7% / 97.0%
of the time, 4.1 / 5.1 mean turns.
On the stock-2B column. It is a fresh run of this harness against
Qwen/Qwen3.5-2B, not the figure the trainer logged at step 0 (which was 0.13-0.16
on full). The two disagree, and the cause is not settled: the behavioural
counters match closely but the fallback-to-last-working-query fires on 7% of
episodes here against 41% there, and an untrained model's score is dominated by
that fallback. The trained models barely use it (2-7%), which is why their
numbers reproduce their training records to within a point or two. The honest
reading is that the 0.290 figure is the one measured the same way as every other
bar in the chart, and that an untrained baseline in this harness carries more
measurement uncertainty than the trained ones do.
Does the published repo match what training measured?
Training could not measure this repo. Its vLLM engine was built from the base model with the checkpoint's weights synced in — a workaround forced by the fact that the raw training export is not loadable by vLLM at all. This repo is the rebuilt, directly-loadable one, so "same model" needed checking rather than assuming.
Both paths were run over the same 300 questions and compared paired (exact McNemar + bootstrap CI), which is the right test here because at n=300 the unpaired noise floor is several points:
| mode | weight-sync path | this repo | paired delta | McNemar p |
|---|---|---|---|---|
full | 0.517 | 0.553 | +0.037, 95% CI [-0.007, +0.080] | 0.13 |
withheld | 0.517 | 0.547 | +0.030, 95% CI [-0.010, +0.070] | 0.18 |
No significant difference in either mode, and the published path never scored lower. Read that as "consistent within about ±5 points", not as proof of equality — failure to reject is not equivalence, and the CI width is what the test can actually exclude.
Two caveats that apply to the absolute numbers regardless:
- The checkpoint was selected on BIRD. Steps 1250-1650 all sit in a
0.51-0.56 band, so treat ~0.55
fullas the plateau rather than any single step as a clean result. - This does not separate from its sibling. The two models differ by ~1 point, well inside the floor. Do not read the training-data comparison off these numbers; run a paired test if you need it.
Numbers are BIRD dev, produced by this project's own harness, and are not leaderboard submissions.
Serving with vLLM
The repo is a complete Qwen3_5ForConditionalGeneration checkpoint, so it loads
directly — no conversion, no --hf-overrides:
vllm serve cwolff/qwen3.5-2b-grpo-bird --max-model-len 16384
from vllm import LLM, SamplingParams
llm = LLM(model="cwolff/qwen3.5-2b-grpo-bird", max_model_len=16384)
out = llm.chat(
[{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "How many singers are there?"}],
SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=2048),
)
generation_config.json stops on both <|im_end|> (248046) and
<|endoftext|> (248044), matching the stop set used during RL rollouts. The
base model's own config lists only <|endoftext|>, which would run past the end
of every assistant turn.
How to prompt it
This model expects an agent loop, and will underperform badly without one.
It was trained under the exec_plan protocol: each assistant turn emits a
single JSON object and nothing else — no prose, no code fences — and your
harness executes it and feeds the result back as the next user turn.
{"tool": "list_tables"}
{"tool": "describe_table", "args": {"table_name": "singer"}}
{"tool": "run_query", "args": {"sql": "SELECT COUNT(*) FROM singer"}}
{"tool": "submit_sql", "args": {"sql": "SELECT COUNT(*) FROM singer"}}
Tools: list_tables, describe_table, foreign_keys, join_path,
sample_rows, run_query, distinct_values, submit_sql. submit_sql ends
the episode. Assistant turns open with a <think> block — keep thinking
enabled; the chat template does this by default.
The exact system prompt matters. It is EXEC_PLAN_TOOL_INSTRUCTIONS in
shared/tools/protocol.py of the SQaLe finetune repo — use that string rather
than paraphrasing the tool list above, which is a summary.
Training
| Base | Qwen/Qwen3.5-2B, stock (no SFT warm start) |
| Algorithm | GRPO, token-level loss, sum/1024 aggregation, no advantage normalisation |
| KL | k3 estimator toward a frozen reference, target 0.005 |
| Clipping | sequence-level importance ratio from vLLM sampler logprobs, 0.2 / 0.28 (DAPO) |
| Batch | 18 questions × 8 candidates = 144 episodes/step, 1 inner epoch |
| LR | 1e-5, cosine, 5% warmup |
| Reward | dense; 3.0 semantic equivalence, F1 partial credit (scale 0.9), 0.1 join bonus, 0.1 consensus bonus |
| Environment | exec_plan, 6 tool turns, 12288-token episode budget, in-memory SQLite |
| Hardware | 3× H100, 3-rank torchrun data parallel, 48 h wall clock |
| Steps | 1350 of a planned 1800 (the run hit its wall-clock limit at ~1660) |
Reward is computed by executing the candidate against a real SQLite database built from the row payloads and comparing to the gold result — not by string match against a reference query.
Limitations
- 2B parameters. It is a small model and will lose to a frontier model on hard multi-join questions.
- Trained and evaluated on SQLite only.
- Trained at a 6-turn budget with a 12288-token episode cap. Longer budgets are untested and truncation reached 9-22% at this one.
- Single-turn "schema in, SQL out" prompting is out of distribution — the
withheldnumbers above are what this model is actually for. - Inherits the base model's vision tower, which is carried over untouched and was never trained or evaluated here. Treat it as a text model.
Provenance
Slurm job 26640987, output grpo1800_bird, checkpoint best_bird/ (step
1350), selected by the trainer on BIRD full official accuracy.
Exported by merging the run's 320 trained language-model tensors with the
base model's vision tower and MTP head, under the base config.json.