SQL RL Pipeline: Qwen2.5-Coder-3B-Instruct (SFT + GRPO)

An end-to-end Reinforcement Learning fine-tuning pipeline for Text-to-SQL generation. We trained Qwen2.5-Coder-3B-Instruct using a two-stage approach on the Spider dataset, achieving near-perfect SQL execution rewards through execution-guided RL.

Performance Benchmarks

By applying Group Relative Policy Optimization (GRPO) with a custom execution-based reward function (evaluating SQL execution results against real SQLite databases), we pushed well beyond what standard supervised fine-tuning achieves.

Execution Accuracy by Stage (Spider Dev Set)

GRPO Training: Reward Progression

The reward function scores 1.0 when the model's generated SQL produces an exact execution result match against the gold query on the real Spider SQLite databases, and 0 otherwise.

Model Details

Pipeline Architecture

Spider Dataset
     |
     v
[1] make_dataset.py        -- Format prompts + gold SQL for SFT and GRPO
[2] sft_train.py           -- Unsloth SFT with LoRA (3 epochs, 2625 steps)
     |
     v
checkpoints/sft_merged     -- Full merged SFT model
     |
     v
[3] grpo_train.py          -- TRL GRPO with execution-based reward (300 steps)
     |
     v
checkpoints/grpo_final_lora
     |
     v
[4] merge_and_upload.py    -- Merge GRPO LoRA onto SFT base, push to HuggingFace
     |
     v
riit3sh/qwen2.5-coder-3b-instruct-spider-sft-grpo  (HuggingFace)

Reward Function

The GRPO reward is execution-guided: each generated SQL query is executed against the actual Spider SQLite database. The reward is:

• 1.0 — execution result exactly matches the gold query result
• 0.0 — wrong result, SQL error, or timeout

No partial credit is given, forcing the model to produce fully correct, executable SQL.

Quickstart

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "riit3sh/qwen2.5-coder-3b-instruct-spider-sft-grpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

schema = "students | id, name, age, gpa | courses | id, name, credits"
question = "What is the average GPA of all students?"

prompt = f"""You are an expert SQL assistant. Given a database schema and a question, write a correct SQL query.

Schema: {schema}
Question: {question}
SQL:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.1, do_sample=True)

sql = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
print(sql)

Serving with FastAPI + vLLM

pip install vllm fastapi uvicorn pydantic
python app.py

curl -X POST http://localhost:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"schema_str": "student | id | name | age", "question": "What is the average age of all students?"}'

Reproducing the Pipeline

git clone https://github.com/riit3sh/sql-rl-pipeline
cd sql-rl-pipeline
pip install -r requirements.txt

# Stage 1: SFT
python scripts/sft_train.py

# Stage 2: GRPO
python scripts/grpo_train.py

# Merge + Upload
python merge_and_upload.py

Built with Unsloth, TRL, and vLLM.