PrismaCoder-3B

PrismaCoder-3B is a Qwen2.5-Coder-3B-Instruct fine-tune specialized for generating and editing Prisma ORM schemas and client code from natural-language instructions. It was trained with a full SFT → DPO → GRPO pipeline, using a verifier (gate) as both the data-quality filter and the reinforcement-learning reward signal.

Baseline comparison — 200-task PrismaBench subset, static V0 gates, greedy decoding, identical prompts:

Run with the same evaluator, same prompt construction, same decoding settings (greedy, max_new_tokens=768) across every model — see Evaluation Methodology.

This 200-task result is consistent with, not a replacement for, the authoritative full-benchmark number: 0.9992 pass@1 (Wilson [0.9953, 0.9999]) on the complete 1,200-task PrismaBench, reported below.

Model Details

• Base model: Qwen/Qwen2.5-Coder-3B-Instruct
• Parameters: ~3B
• Fine-tuning method: QLoRA (r=16, α=32, all linear projections) at each stage, LoRA weights merged into the base model between stages
• Training stages: SFT → DPO → GRPO (online RL, verifier reward) → iterative self-play RL round (attempted; see Limitations)
• Precision: 4-bit NF4 (bitsandbytes) during training, fp16/bf16 merged weights for release
• License: apache-2.0 (matches the base model)

Training Procedure

All stages used seed 20260830 and ran on a single Kaggle T4 GPU (16GB) with gradient checkpointing and paged 8-bit AdamW.

Reward/gate design: every stage is filtered or scored by the same static verifier (LightValidator, "V0" in the project's internal gate numbering): it checks that generated Prisma schema/client code parses, has balanced code fences, and that referenced models/fields are actually grounded in the provided schema. It does not check that the generated client code compiles or executes correctly against a real database — that requires the full gate chain (V1–V5, Node-based), which did not run in this training environment. See Limitations.

Training Data

• Corpus: PrismaForge, a two-track dataset — a small mined track (real schema.prisma files and call sites crawled from permissively-licensed public repos) and a larger synthetic track (mechanically generated task/schema pairs across families, domains, and difficulty levels).
• Scale actually used for this model: ~3,000 of ~17,500 available finalized training records (MAX_TRAIN_SAMPLES=3000). This is smaller than the project's full-scale design target (120,000 synthetic records); results should be read as a proof-of-concept at this data scale, not the ceiling of the method.
• Contamination firewall: training records whose provenance hash matches a PrismaBench benchmark task are excluded from the training splits before SFT.

Practical implication: always run prisma validate (or the full V1–V5 gate chain, not yet run against this model — see Limitations) on any output before using it. Do not treat a high pass@1 score alone as a guarantee of correct, usable Prisma code, particularly for relation fields.

Evaluation

Benchmark: PrismaBench, a 1,200-task benchmark (categories T1–T12, difficulty levels L1–L3) built by the same project, with canary strings and a provenance registry used for the contamination firewall above.

Gate mode: static V0 gates only (static evaluation mode (no Node or --full-gates not passed) — printed by the evaluator on every run in this environment). This distinction matters: V0-only pass@1 measures well-formed, schema-grounded output; full-gate pass@1 (not yet run) would measure whether the generated code actually compiles, migrates, and executes correctly.

Full PrismaBench, N=1,200, static V0 gates — checkpoint: merged/grpo

The low exact-match alongside near-perfect pass@1 is expected, not a red flag: pass@1 checks gate-level correctness (parses, balanced fences, schema-grounded references), not verbatim agreement with a single reference solution — code that is correct but phrased differently from the reference scores low on exact-match by design.

Per-category pass@1 (1,200 tasks):

Error taxonomy: 1 failure classified E4; no other error classes observed across 1,200 tasks.

Which checkpoint this reflects: RL round 2 also completed successfully in training (self-play produced enough separable pairs this run, unlike an earlier attempt) and reached the same 0.975 proxy score as GRPO, but the 1,200-task benchmark above was run against the GRPO checkpoint (merged/grpo), not merged/rl2. The RL round 2 checkpoint has not yet been scored on the full benchmark — see Next Steps in the project repo before treating GRPO as the final, best checkpoint.

Baseline comparison: the only benchmark comparison currently valid for this card is against the untrained Qwen2.5-Coder-3B-Instruct base model, scored on the identical 1,200-task set (0.700 proxy pass@1 pre-training; full-benchmark base-model pass@1 not separately run). Comparisons against other released coding models are not included here because they have not been run through this same evaluator on this same held-out task set — a fair comparison requires that, and until it's done, any such claim on this card would be a number this project didn't actually measure.

Intended Use

• Generating and editing Prisma schema definitions and Prisma Client TypeScript code from natural-language instructions.
• Assistance with schema migrations, CRUD operations, and query composition against an existing Prisma schema.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sandeeprdy1729/PrismaCoder-3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a Prisma ORM code generation assistant."},
    {"role": "user", "content": "Add a `Comment` model related to an existing `Post` model, with a body field and a createdAt timestamp."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=640, do_sample=False)  # greedy - matches published eval
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Always validate generated schemas before use, given the known @relation issue above:

npx prisma validate --schema=./generated_schema.prisma

A GGUF export (PrismaCoder-3B-f16.gguf) is also provided for llama.cpp/Ollama-based inference.

Training Infrastructure

Trained on Kaggle (single NVIDIA T4, 16GB), 4-bit QLoRA throughout, with step-level checkpointing every 10–25 steps and cross-session resume to fit within Kaggle's 12-hour session limit.

Citation

If you use this model, please cite the base model and note this fine-tune:

@misc{prismacoder3b,
  title  = {PrismaCoder-3B: QLoRA/DPO/GRPO fine-tune of Qwen2.5-Coder-3B for Prisma ORM code generation},
  author = {Thummala, Sandeep},
  year   = {2026},
  url    = {https://huggingface.co/sandeeprdy1729/PrismaCoder-3B}
}

Base model: Qwen2.5-Coder-3B-Instruct.