SmolLM2-135M-text2cypher
A fine-tune of HuggingFaceTB/SmolLM2-135M-Instruct
that turns a natural-language question + a graph schema into a Cypher query.
Trained with LoRA on RomanTeucher/text2cypher-curated
(1000 train / 75 val / 50 test); the adapters are merged into the base weights,
so this is a standalone full model that loads like any HF checkpoint — no PEFT needed.
This model is a submission for an interview.
Usage
The model expects the same ChatML prompt it was trained on: a fixed system message
plus a user turn of Schema: + Question:.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "rizwan261/smollm2-135m-text2cypher"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
SYSTEM = ("You are a text-to-Cypher engine for Neo4j. Given a graph schema and a "
"question, output a single valid Cypher query that answers the question. "
"Use only labels, relationship types and properties that appear in the "
"schema. Output the Cypher query and nothing else: no explanation, no "
"comments, no markdown code fences.")
schema = "Graph schema: Relevant node labels and their properties (with datatypes) are:\nUpdateDate {update_date: DATE}"
question = "Which nodes are connected to UpdateDate where update_date is 2008-01-29, and also to another node?"
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Schema:\n{schema}\n\nQuestion:\n{question}"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Training
LoRA (r=16, α=32, dropout=0.05) on the attention + MLP projections, completion-only loss (the prompt is masked, loss is on the target Cypher only), then merged.
| epochs | 20 |
| learning rate | 5e-5, cosine, 5% warmup |
| effective batch | 16 (bs 4 × grad-accum 4) |
| max length | 1024 |
| best val loss | ~0.32 (by eval_loss) |
Evaluation
Test split (n=50), greedy decoding. Translation / structural metrics over the whole split — surface proxies for correctness:
| metric | base | this model |
|---|---|---|
| exact_match | 0.00 | 0.26 |
| structural_f1 | 0.18 | 0.81 |
| google_bleu | 0.07 | 0.70 |
| well_formed | 0.00 | 0.98 |
Execution-based evaluation against the live Neo4j demo databases, on the 18/50 samples that carry a database reference — predicted and gold queries are run and their result sets compared. This is the honest measure, and each stricter layer peels back the previous one's optimism:
| how strict | metric | base | this model |
|---|---|---|---|
| real parser + live schema | CyVer KG-Valid-Query | 0.00 | 0.61 |
| runs on the DB | query executes | 0.00 | 0.72 |
| returns the right rows | exec ExactMatch (non-trivial) | 0.00 | 0.00 |
The surface metrics jump, but execution accuracy stays at zero: the model writes well-formed, schema-valid, runnable Cypher that returns the wrong data.
Limitations
A 135M model produces Cypher that often looks right but isn't: only ~26% are exact matches, and non-trivial execution accuracy is ~0 — well-formed, runnable queries that return the wrong rows. It reliably gets node labels, clauses and the query skeleton, but struggles with same-node/co-reference patterns, multi-stage aggregation, and occasionally hallucinates functions. Execution numbers come from a small (n=18) subset run against live demo databases, so they can drift.
License
Apache-2.0, inherited from the SmolLM2 base model.