lumina-tsql-8b

What this is

Lumina T-SQL 8B turns natural-language questions about a wholesale-distribution ERP database into executable T-SQL (SQL Server dialect). It is a preference-tuned Qwen/Qwen3-8B, aligned with execution-grounded DPO on ERPBench Distribution, our open benchmark built on an original 60-table distribution schema (luminadist) seeded with 2.56M rows.

Three things make it different from a generic SQL fine-tune:

1. Execution is the only teacher. Preference pairs were built by sampling the model's own candidates, running every one against a live seeded SQL Server, and grading by result-set match. Chosen answers are the model's own correct queries; rejected answers are its executable-but-wrong hard negatives. No human ranking, no synthetic judge.
2. It keeps its instruction-following. Conventional SFT on narrow SQL data cratered the base model's ability to obey output contracts (return exactly these columns, order the rows like this). Preference tuning against the stock reference preserved it and then improved it: 79.0% on instruction-heavy single-table questions vs 60.5% for the base model.
3. It ships with schema-constrained decoding. constrained_decoding.py plus schema_catalog.json (both in this repo) let you lint generated SQL against the schema (check_sql) or mask logits during generation so the model cannot emit a table or column that does not exist.

Results

Execution-match accuracy on ERPBench Distribution, eval v2.1 (greedy decoding, thinking disabled, one attempt, no retries, no constrained decoding):

With execution-based voting at k=16 (sample, run every candidate, return the plurality result set), the same weights score 46.8 on dev and 35.5 on test, and invalid SQL becomes rare: execution errors fall to 20 out of 500 dev items, a 96% valid-query rate. vote_infer.py in this repo is that recipe, and it is the recommended way to deploy the model.

Per-tier on dev (T1 single table through T5 recursive CTEs and PIVOT):

Reading these honestly:

• The private holdout was never published, never used in any training stage, and drew from tables excluded from training data. The largest gain landing there is the strongest evidence the improvement is real skill, not leakage.
• The test split is deliberately harder than dev: it holds out entire compositional join families that never appear in training. Stock scores 20.5 / 20.4 on test / holdout (a 0.1 gap), which shows the two splits are fairly calibrated; the difficulty is in the tasks, not the split.
• Invalid SQL dropped alongside the accuracy gain: 117 execution errors on dev vs 145 for stock.
• The headline table is single-shot with no scaffolding, which understates deployed accuracy. Execution voting converts much of the sampling headroom into answers: +2.6 on dev and +8.7 on test over greedy, and it is where the reliability gain lives, taking invalid SQL from 100 dev items to 20. Two cheaper scaffolds do not help. A one-round check_sql correction loop and the mechanical repair_sql column fixer both land within half a point of greedy, because fixing an identifier turns an error into an executable but wrong query rather than a right one. Preferring candidates whose result columns match the ones the question names showed no gain over plain plurality either.
• Failure anatomy from 16,000 graded samples: 74% of execution errors are invalid column names; invalid table names are nearly extinct (36 occurrences in 8,000 dev samples). The model has effectively internalized the table namespace; columns are the open frontier.
• The model reads the schema you give it. In a renamed-table probe (schema tables renamed inside the prompt so the in-context schema contradicts training), 85% of samples followed the prompt schema and only 7.8% reverted to memorized names.

How it was trained

1. SFT was tried first and rejected on evidence. Three LoRA SFT cycles on 4.7K verified pairs improved hard tiers but collapsed instruction-following (single-table accuracy fell from 60.5 to below 29). Every cycle scored below the stock baseline overall.

2. DPO round 1 diagnosed the failure mode. Beta 0.1 with 66% of chosen answers taken from gold references (off-policy) over-optimized: reward margins exploded, generation quality degraded, score fell to 22.8.

3. DPO round 2 is this model. 447 strictly on-policy pairs (chosen must be the model's own correct sample, rejected is its own executable-but-wrong candidate, both verified by execution), LoRA rank 64, beta 0.3, one epoch against the frozen stock reference. Result: gains on every split with instruction-following improved, not sacrificed.

4. Round 3 was trained and rejected by the same gate. A further iteration on 636 pairs sampled from the round-2 policy raised easy-tier accuracy but regressed the hard tiers below the round-2 score, so it was not shipped. Iterated on-policy DPO sharpened what the model already did well at the cost of diversity where it is weak.

5. Rejection-sampling SFT is what broke the plateau, and it is the released model. Across four preference rounds, join accuracy never moved, even when joins made up 29% and then 47% of the pairs. So the objective changed rather than the data mix: sample the policy over training prompts, execute every candidate, keep only the ones whose result set matches gold, and train on those as ordinary supervised examples. 3,139 verified-correct samples, one epoch at 2e-5 continuing from the round-2 weights. Every tier improved, joins included, and execution errors fell further. Ranking candidates against each other never taught composition; amplifying verified-correct outputs did.

The pipeline is fully mechanical and rerunnable: sample k=8 at temperature 0.8, execute all candidates, keep contrastive pairs, train, evaluate against the gate. Each round costs a few GPU-hours on one H100, and failed rounds are caught before anything is published.

Usage

The model expects the ERPBench system prompt (schema included). It ships in this repo as system_prompt.txt. Disable thinking for this task.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Lumina-ERP/lumina-tsql-8b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

system = open("system_prompt.txt", encoding="utf-8").read()
question = ("List every warehouse code and its city. "
            "Return exactly these columns: warehouse_code, city.")

messages = [{"role": "system", "content": system},
            {"role": "user", "content": question}]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

vLLM

vllm serve Lumina-ERP/lumina-tsql-8b --max-model-len 16384

# request body extras
{"temperature": 0, "chat_template_kwargs": {"enable_thinking": false}}

Schema-constrained decoding

from constrained_decoding import SchemaCatalog, check_sql

catalog = SchemaCatalog.load("schema_catalog.json")
report = check_sql(sql, catalog)
if not report.ok:
    print(report.problems)      # unknown tables / columns, with suggestions

SchemaLogitsProcessor in the same file plugs into vLLM for hard masking during generation. See example_infer.py for an end-to-end script.

Evaluation and reproduction

Grading is execution match: the query must run against the seeded luminadist database and return the gold result set (column count enforced; row order enforced only when the gold query has a top-level ORDER BY; cells canonicalized; 35s timeout; 200,000-row cap). The dataset repo ships the schema, the deterministic seed generator, a docker compose file for SQL Server 2022, and eval/eval.py, which reproduces the numbers above against any OpenAI-compatible endpoint.

The 250-item private holdout is retained by Lumina ERP as a contamination tripwire and is scored on request for serious submissions.

Limitations

• Specialized for T-SQL against the ERPBench distribution schema and prompt contract. It is not a general SQL assistant, and other dialects (MySQL, Postgres) are out of scope.
• Multi-join analytical queries (T2 through T4) remain hard: absolute accuracy on the compositional test split is 22.5%. We publish that number because honest task difficulty is the point of the benchmark.
• The model can still produce wrong-but-executable queries. For production use, pair it with the shipped check_sql lint and result review.
• Trained and evaluated in English.

About

Built by Lumina ERP. The schema, data, questions, and preference pairs are fully synthetic and original (cleanroom); no customer data or proprietary ERP schemas were used in any stage. Base model by the Qwen team, Apache 2.0.