sqlpup-394m-sft-grpo
The SFT model above plus a GRPO stage over an execution reward. 23.51% execution accuracy on BIRD dev.
Pretrained from scratch on 9.67B tokens, supervised fine-tuned, then trained with GRPO against a three-tier execution reward on prompts the supervised stage did not sample. The gain over the supervised checkpoint is significant in one decoding configuration of three, which the project reports rather than reducing to a single number.
Code, configs, and the per-run artifacts behind every number here: github.com/shivenkk/sqlpup.
What it is
A 394M-parameter decoder trained from scratch for text-to-SQL: no pretrained initialisation, no distillation from a larger model. The corpus (9.67B tokens) and the 32,768-entry byte-level BPE tokenizer were built for this task.
Architecture: Llama-style, 1024 wide by 32 layers, grouped-query attention with 16 query heads and 4 key/value heads, RoPE, SwiGLU, RMSNorm, 2048-token context.
Measured accuracy
Execution accuracy on the BIRD development split, all 1534 questions:
| decoding | sqlpup-394m-sft | this model |
|---|---|---|
| greedy | 17.41% | 18.12% |
| + schema compaction | 17.86% | 19.23% |
| + compaction, 7-sample voting | 22.84% | 23.51% |
The voting rows are means over sampling seeds 0/101/202. Absolute accuracy is far from frontier systems; the point of the project was a controlled measurement of what from-scratch pretraining at this scale buys, not a leaderboard score.
Prompt format
The model expects the schema's CREATE TABLE statements verbatim, then the
question, then BIRD's evidence string. It emits a short comment block naming the
tables and columns it intends to use, then the SQL. Use the accompanying code so
the format and the SQL extraction match what the numbers were measured with.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("shivenkk/sqlpup-394m-sft-grpo")
tok = AutoTokenizer.from_pretrained("shivenkk/sqlpup-394m-sft-grpo")
For evaluation, prefer the project's own harness, which handles the prompt spec, over-context schema compaction, and execution-guided voting:
python -m sqlpup.cli eval predict --model-dir <dir> --examples dev.json \
--db-root dev_databases --out-dir out --compact-overflow --self-consistency 7
Limitations
- 2048-token context. Large schemas need the compaction pass to fit.
- Trained and measured on BIRD only. No test-split number, no second dataset.
- Roughly 1% of dev questions produce no parseable SQL in the voting configuration, recorded as errors rather than dropped.
Training data
FineWeb-Edu, StarCoder, the Python subset of The Stack, SynSQL-2.5M, and SchemaPile, deduplicated and decontaminated against the evaluation sets.