Distil-Qwen3-4B-Text2SQL-GGUF-4bit
4-bit quantized GGUF version of distil-qwen3-4b-text2sql for efficient local inference. Only 2.5GB - runs on most laptops and edge devices.
Results
| Metric | DeepSeek-V3 (Teacher) | Qwen3-4B (Base) | This Model |
|---|
| LLM-as-a-Judge | 80% | 62% | 80% |
| Exact Match | 48% | 16% | 60% |
| ROUGE | 87.6% | 84.2% | 89.5% |
Quick Start with Ollama
1. Download the model
git lfs install
git clone https://huggingface.co/distil-labs/distil-qwen3-4b-text2sql-gguf-4bit
cd distil-qwen3-4b-text2sql-gguf-4bit
2. Create and run the model
# Create the Ollama model (Modelfile is included)
ollama create distil-qwen3-4b-text2sql -f Modelfile
# Run the model
ollama run distil-qwen3-4b-text2sql
3. Test it
>>> Schema:
... CREATE TABLE employees (id INTEGER PRIMARY KEY, name TEXT, department TEXT, salary INTEGER);
...
... Question: How many employees earn more than 50000?
SELECT COUNT(*) FROM employees WHERE salary > 50000;
Usage with Python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="EMPTY")
schema = """CREATE TABLE employees (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
department TEXT,
salary INTEGER
);"""
question = "How many employees earn more than 50000?"
response = client.chat.completions.create(
model="distil-qwen3-4b-text2sql",
messages=[
{
"role": "system",
"content": """You are given a database schema and a natural language question. Generate the SQL query that answers the question.
Rules:
- Use only tables and columns from the provided schema
- Use uppercase SQL keywords (SELECT, FROM, WHERE, etc.)
- Use SQLite-compatible syntax
- Output only the SQL query, no explanations"""
},
{
"role": "user",
"content": f"Schema:\n{schema}\n\nQuestion: {question}"
}
],
temperature=0
)
print(response.choices[0].message.content)
# Output: SELECT COUNT(*) FROM employees WHERE salary > 50000;
Model Details
Why Use This Version?
- Small size: 2.5GB vs 15GB (full GGUF) or 8GB (safetensors)
- Fast inference: Optimized for CPU and consumer GPUs
- Same accuracy: Quantization has minimal impact on Text2SQL quality
- Easy setup: Works with Ollama out of the box
Related Models
Supported SQL Features
- Simple: SELECT, WHERE, COUNT, SUM, AVG, MAX, MIN
- Medium: JOIN, GROUP BY, HAVING, ORDER BY, LIMIT
- Complex: Subqueries, multiple JOINs, UNION
License
This model is released under the Apache 2.0 license.
Links