Qwen3 1.7B Salesforce Fine-tuned SOQL
Fine-tuned from Qwen/Qwen3-1.7B for Salesforce schema reasoning, grounded SOQL workflows, and CRM agent tool routing.
🚀 Benchmark at a glance
| Model | Dev | Challenge |
|---|---|---|
| Qwen3-1.7B Base | 50.9% | 55.0% |
| Salesforce Fine-tuned SOQL | 83.0% | 85.0% |
+32.1 percentage points on Dev · +30.0 percentage points on Challenge
Detailed benchmark
| Model / Suite | Tool Routing | Grounding | Hallucination Guard |
|---|---|---|---|
| Base / Dev | 86.8% | 60.4% | 100.0% |
| Fine-tuned / Dev | 96.2% | 86.8% | 98.1% |
| Base / Challenge | 85.0% | 70.0% | 100.0% |
| Fine-tuned / Challenge | 100.0% | 85.0% | 100.0% |
Evaluation uses 53 frozen development cases and 20 challenge cases.
Inference was deterministic withdo_sample=Falseandenable_thinking=False.
A fine-tuned Qwen3-1.7B model designed for Salesforce, SOQL, CRM reasoning, and local-first agent workflows.
Repository: ravinarayanan/Qwen3-1.7B-Salesforce-Finetuned-SOQL
What this model is for
This model is optimized to:
- understand Salesforce-oriented user requests
- decide when Salesforce metadata must be retrieved
- select tools such as
search_schema,describe_object,run_soql, andanalyze_data - generate grounded, read-only SOQL
- reason about Salesforce parent and child relationships
- avoid inventing Salesforce objects, fields, relationships, records, or filter values
- retain general assistant abilities for writing, summarization, and explanation
Runtime Salesforce metadata should remain the source of truth.
Base model
Qwen/Qwen3-1.7B
Current agent workflows use enable_thinking=False for deterministic non-thinking tool routing.
Benchmark Results
The frozen evaluation benchmark contains:
- 53 development cases
- 20 challenge cases
- 73 total scenarios
It covers schema retrieval, field grounding, relationships, aggregation, query semantics, CRM-grounded writing, general assistant behavior, and hallucination resistance.
Overall accuracy
| Model | Dev | Challenge |
|---|---|---|
| Qwen3-1.7B Base | 27/53 (50.9%) | 11/20 (55.0%) |
| Salesforce Fine-tuned SOQL | 44/53 (83.0%) | 17/20 (85.0%) |
Improvement over base:
- +32.1 percentage points on Dev
- +30.0 percentage points on Challenge
Detailed metrics
| Model / Suite | Tool Routing | Grounding | Hallucination Guard |
|---|---|---|---|
| Base / Dev | 86.8% | 60.4% | 100.0% |
| Fine-tuned / Dev | 96.2% | 86.8% | 98.1% |
| Base / Challenge | 85.0% | 70.0% | 100.0% |
| Fine-tuned / Challenge | 100.0% | 85.0% | 100.0% |
Remaining weak areas
- known-object / unknown-field routing
- some parent relationship cases
- aggregation edge cases
- some query-semantics cases
Benchmarks were run deterministically with do_sample=False and enable_thinking=False.
Salesforce grounding principles
- Never invent Salesforce schema.
- Retrieve schema when object or field information is unknown.
- Use relationship names exactly as returned by Salesforce metadata.
- Never derive custom relationship API names from lookup field names.
- Generate read-only SOQL.
- Never invent filter values that were not requested.
- Base Salesforce analysis only on retrieved records or deterministic calculations.
Tool-oriented behavior
Typical tools:
search_schema
describe_object
run_soql
analyze_data
Example user request:
Show me all renewal risk records.
Expected behavior:
{
"action": "search_schema",
"arguments": {
"term": "renewal risk"
}
}
The model should not assume that an object such as Renewal_Risk__c exists.
Recommended architecture
User
|
Local LLM
|
Agent Controller
|
+-- search_schema
+-- describe_object
+-- run_soql
+-- analyze_data
|
Deterministic SOQL validation
|
Salesforce REST API
Model format
This repository contains the merged Transformers checkpoint.
Browser/WebGPU quantization is being evaluated separately and is not included in this checkpoint.
Limitations
Production applications should still:
- validate SOQL deterministically
- enforce read-only execution
- respect object and field permissions
- validate tool arguments
- cap query limits
- treat runtime Salesforce metadata as authoritative
Disclaimer
This is an independently developed model for Salesforce-related developer and agent workflows.
Salesforce is a trademark of Salesforce, Inc.
This project is not affiliated with or endorsed by Salesforce.