🔮 Llama-3.2-3B-Instruct Text-to-SQL

A fine-tuned version of unsloth/Llama-3.2-3B-Instruct-bnb-4bit for generating SQL queries from natural language questions and database DDL schemas.

Model Summary

Evaluation Results

Evaluated on the first 200 samples of the gretelai/synthetic_text_to_sql test split using greedy decoding. ROUGE F-measures reported.

Metric interpretation:

• ROUGE-1 (unigram overlap) reflects accurate retrieval of schema identifiers and SQL keywords.
• ROUGE-2 (bigram overlap) captures structural alignment of consecutive SQL constructs (e.g. GROUP BY, ORDER BY).
• ROUGE-L (longest common subsequence) tracks overall query flow including nested clauses and join ordering.

Note: ROUGE measures lexical overlap, not SQL executability. A query may score slightly lower due to stylistic differences (alias names, join ordering) while still being functionally equivalent. See Limitations.

How to Use

The model weights are fully merged in 16-bit precision and load with standard transformers or unsloth.

Prompt Format

Always use this exact template — the model was trained on it:

###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query

(a) Standard transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "A-Kishore/llama-3.2-3b-text2sql"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = """###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query
"""

sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
sql_prompt = "What is the average salary per department?"

inputs = tokenizer(
    prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    use_cache=True,
    pad_token_id=tokenizer.eos_token_id
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
sql = result.split("### SQL Query")[-1].strip()
print(sql)
# SELECT department, AVG(salary) FROM employees GROUP BY department;

(b) Unsloth Fast Inference

import torch
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="A-Kishore/llama-3.2-3b-text2sql",
    max_seq_length=768,
    dtype=torch.float16,
    load_in_4bit=False,
)
FastLanguageModel.for_inference(model)

prompt = """###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query
"""

sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
sql_prompt = "What is the average salary per department?"

inputs = tokenizer(
    prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=None,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
sql = result.split("### SQL Query")[-1].strip()
print(sql)

Training Details

Dataset

• Source: gretelai/synthetic_text_to_sql
• Training split: 50,000 shuffled samples from train
• Evaluation split: First 200 samples from test

LoRA Configuration

LoRA freezes the base model weights and injects trainable rank-decomposition matrices into all attention and MLP projections. Only ~0.75% of parameters are updated, dramatically reducing VRAM usage and preventing catastrophic forgetting.

Hyperparameters

Training was accelerated using the unsloth library, which provides optimized GPU kernels for 4-bit quantized training (~2× faster than standard configurations).

Repository

Training and evaluation code: a-kishore-dev/llama-text2sql-finetune

Notebooks included:

• Text_to_SQL_Finetuning.ipynb — dataset prep, LoRA config, training, export
• evaluate_model.ipynb — ROUGE evaluation comparing base vs fine-tuned

Limitations

• SQL executability: ROUGE is a lexical proxy. High ROUGE does not guarantee a query will execute or return logically correct results. A query with different aliases or reordered joins may score lower despite being equivalent.
• Out-of-distribution schemas: Performance degrades on high-cardinality databases, deeply nested subqueries, or DDL patterns that diverge significantly from the training distribution.
• Single epoch: The model was trained for one epoch on 50k samples. Further training may improve generalization.

License

The model adapter is released under Apache 2.0. The underlying base model is governed by the Meta Llama 3 Community License Agreement. Users must comply with both.

Acknowledgements

• Unsloth — optimized kernels for 4-bit training and sequence packing
• Hugging Face — trl (SFTTrainer) and transformers
• Meta AI — Llama 3.2 open weights
• Gretel AI — synthetic Text-to-SQL dataset

Author

A-Kishore · GitHub · HuggingFace