SQL-to-Text (Salesforce/codet5p-220m)
Salesforce/codet5p-220m fine-tuned to explain a SQL query in plain English.
The direction is SQL -> natural language: the model takes a query (and, optionally, the DDL of the tables it touches) and returns a sentence describing what that query does. It does not generate SQL from a question.
Prompt format
Inputs follow one fixed template; training, evaluation and inference all build
it with the same function, so they cannot drift apart. The schema block is
dropped when no DDL is supplied, and when it is supplied only CREATE TABLE ... statements are kept.
Explain the following SQL query.
Schema:
<CREATE TABLE statements>
SQL:
<the query>
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = "thealper2/codet5p-sql2text"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
schema = "CREATE TABLE employees (id INT, name TEXT, salary INT, dept_id INT);"
sql = "SELECT dept_id, AVG(salary) FROM employees GROUP BY dept_id;"
prompt = f"Explain the following SQL query.\n\nSchema:\n{schema}\n\nSQL:\n{sql}"
inputs = tokenizer(
prompt,
return_tensors="pt",
truncation=True,
max_length=256,
)
outputs = model.generate(
**inputs,
num_beams=4,
max_new_tokens=128,
min_new_tokens=5,
early_stopping=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training data
gretelai/synthetic_text_to_sql, mapping
sql + sql_context
to sql_explanation.
Preprocessing drops rows that are too short to be a real explanation, removes exact duplicates and duplicate inputs, and removes any training row whose input also appears in the official test split, so the reported test scores are not inflated by leakage. The validation split is 3% of the cleaned train split (seed 42).
Sequence lengths were chosen from the measured token-length distribution: source 256 tokens, target 128 tokens.
Training procedure
| Hyper-parameter | Value |
|---|---|
| Epochs | 3.00 |
| Learning rate | 0.0001 |
| LR schedule | linear |
| Warmup ratio | 0.0500 |
| Weight decay | 0.0100 |
| Optimiser | adamw_torch |
| Per-device train batch size | 16 |
| Gradient accumulation | 4 |
| Max gradient norm | 1.00 |
| Model selection | eval_rougeL |
| Seed | 42 |
| Effective batch size | 64 |
Trained on a single NVIDIA GeForce RTX 5060 Ti (15.9 GB) with torch 2.11.0+cu128, bf16 mixed precision.
Wall-clock training time: 84 minutes.
Evaluation
Scored by evaluate.py on the full splits with beam search (num_beams=4).
Generation quality
| Metric | Validation | Test | _meta |
|---|---|---|---|
| Examples | 2989 | 5850 | - |
| BLEU | 33.65 | 33.10 | - |
| ROUGE-1 | 67.15 | 66.86 | - |
| ROUGE-2 | 45.76 | 45.11 | - |
| ROUGE-L | 57.58 | 57.06 | - |
| Mean generated length | 36.37 | 35.69 | - |
| Loss | 0.5829 | 0.5923 | - |
SQL-aware faithfulness
Recall metrics ask whether the explanation mentions what the query actually does; the rate metrics are error rates, where lower is better -- they measure claims the query does not support.
| Metric | Validation | Test |
|---|---|---|
| Examples | 2989 | 5850 |
| Operation recall | 98.53 | 98.40 |
| Aggregation recall | 98.72 | 98.57 |
| Join mention recall | 98.21 | 98.74 |
| Join table coverage | 98.34 | 98.56 |
| Condition column coverage | 82.98 | 81.99 |
| Condition value coverage | 86.18 | 87.24 |
| Operation over-claim rate | 5.47 | 5.29 |
| Unsupported number rate | 3.98 | 3.18 |
| Unsupported quoted-string rate | 3.20 | 3.18 |
| Unsupported entity rate | 1.38 | 1.72 |
Limitations
- Trained on synthetic queries and synthetic explanations, so the phrasing reflects that generator's style rather than how a particular team documents its own queries.
- Explanations are grounded in the query text, not in the data: the model cannot know what a column means beyond its name.
- Condition coverage is the weakest area -- long
WHEREclauses lose some columns and literals -- so an explanation may describe a filter less precisely than the query applies it. Do not rely on it as an audit of what a query returns. - Inputs are truncated past the configured source length, so very large schemas are only partially visible to the model.
- English only.
Reproduction
make preprocess
make train
make evaluate
Base model: Salesforce/codet5p-220m.