Qwen3-Embedding-0.6B-GraphQL

A small, self-hosted embedding model for retrieving GraphQL Type.field coordinates from natural-language questions. It is designed for schema retrieval in AI agent pipelines, where the full schema is too large to place in context for every request.

Real schemas reuse field names extensively. Many types expose fields such as title, author, createdAt, state, or status, so matching the field name alone is not enough; retrieval must also identify the correct owner type.

This model fine-tunes Qwen/Qwen3-Embedding-0.6B for that owner-type disambiguation task. At 0.6B parameters, it can run locally on CPU or alongside an agent's primary model.

Benchmark

The GraphQL AI Working Group benchmark evaluates 816 natural-language queries across GitHub, GitLab, Linear, Shopify, and the Singapore Open Data API. Field-level recall against the ground-truth Type.field coordinate:

The fine-tune leads field-level retrieval at both cutoffs and improves recall@50 by 5.1 percentage points over its base model. At the owner-type level, text-embedding-3-large leads at 87.8% recall@50; the fine-tune reaches 85.6%, ahead of the base model at 78.6% and text-embedding-3-small at 80.8%.

The model ships as SentenceTransformers weights and GGUF builds for llama.cpp and Ollama.

Important: how you format the corpus matters as much as the model. Use SDL snippets or dot_plus_gloss formatting for best results. See Embedding style comparison for details.

Inference

SentenceTransformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("xthor/Qwen3-Embedding-0.6B-GraphQL")

query = "When did I place this order?"
# coordinates of Type.field pairs
coords = [
    "Order.createdAt",
    "Order.completedAt",
    "Order.updatedAt",
]

q = model.encode(query, prompt_name="query")
c = model.encode(coords, prompt_name="document")
scores = (q @ c.T).tolist()

for coord, score in sorted(zip(coords, scores), key=lambda x: -x[1]):
    print(f"{score:.3f}  {coord}")

The fine-tuned model ranks Order.createdAt first; the un-tuned base model ranks Order.completedAt first. This is a small sanity check using readable bare coordinates. For production retrieval, use the richer SDL or gloss formatting described below.

Two prompts are wired into the model and must be used for best results:

• prompt_name="query" for natural-language questions
• prompt_name="document" for GraphQL coordinate descriptions in the corpus

Ollama

# pull one quantization (Q8_0 is a good default: near-lossless, ~650 MB)
hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./model-q8_0.gguf
EOF
ollama create qwen3-graphql-embedder -f Modelfile

# OpenAI-compatible embeddings endpoint
curl -s http://localhost:11434/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-graphql-embedder","input":"When did I place this order?"}' \
  | jq '.data[0].embedding'

llama.cpp

hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

./llama-server -m model-q8_0.gguf --embedding --port 8080
# POST http://localhost:8080/embedding   { "content": "..." }

Available GGUF quantizations

Results

223 held-out test queries · 28,893-coordinate corpus · 30% real SDLs (GitHub GHES, Saleor, Shopify, AniList) never seen in training.

Where the lift comes from

Direct questions ("has my package shipped?", "what's my total?") are already handled well by the base model. The gains come from indirect questions where the user names a concept rather than a field. Those require owner-type reasoning, and that's where the base model falls behind.

Small inference example

"When did I place this order?"

Against the bare coordinates Order.createdAt, Order.completedAt, and Order.updatedAt, the base model ranks Order.completedAt first. The fine-tuned model ranks the correct coordinate, Order.createdAt, first and opens a clearer margin:

Every candidate is a plausible timestamp on the same order. The model must distinguish when the order was placed from when it was completed or last updated. This three-candidate example is an illustrative sanity check; the aggregate results above measure retrieval quality over the full held-out corpus.

Known limitations

1. Formatting sensitivity. With raw dot notation (Type.field), the fine-tune's R@1 is only 0.308 on the GitHub schema. Always use sdl, dot_plus_gloss, or natural formatting for the corpus.

2. Same-owner wrong-field rate. same_owner_wrong_field_rate@1 rose from 0.063 to 0.103. The model picks the right owner type more often but occasionally lands on the wrong field within that type. The training signal rewards owner disambiguation; within-owner field disambiguation isn't targeted. The next iteration will add competition sets that share owner and differ by field.

3. Tail regression with raw dot notation. When using raw dot notation, the fine-tune's P95 rank (404) is worse than the base model's (123). The model becomes more confident: it either ranks the correct answer first or misses much harder. This is fully mitigated by using sdl (P95 40) or dot_plus_gloss (P95 41) formatting.

4. Indirect queries. Queries that don't name or allude to the owner type (e.g., "get the README" → Repository.object) remain hard for both models. The fine-tune does not improve on these.

How you format the corpus matters

How you turn each Type.field coordinate into text before embedding it affects retrieval more than the fine-tune does. The benchmark below compares twelve formats on the GitHub GraphQL schema (52 held-out queries):

Use one of these two. They tie at the top:

# sdl: if you parse the schema (MRR 0.723)
type PullRequest { baseRefName: String! }

# dot_plus_gloss: string-only, no parsing needed (MRR 0.715)
PullRequest.baseRefName — the base ref name of a pull request

The cheap string-only gloss costs almost nothing versus full schema parsing, so reach for dot_plus_gloss unless you already have parsed types on hand. Whatever you do, don't embed raw Type.field identifiers. With dot formatting, MRR drops to 0.393 and the worst-case rank blows out 10x. The owner type is what carries the signal: drop it entirely and retrieval collapses to MRR ~0.05.

Full results

Each format is one way of rendering PullRequest.baseRefName into text before embedding (the example column shows exactly what). P95 is the 95th-percentile rank, i.e. how badly the worst queries rank. Lower is better.

Training

Both: --max-seq-length 256, 4 hard negatives per anchor, bf16, full fine-tune (no LoRA), single H100. Published checkpoint: qwen3-e3.

Dataset

Built from 7,626 raw seed pairs via world-leakage, per-row strict-leakage, and family-level semantic-dedup filters. The strict-leakage filter is aggressive on real-SDL queries, which is why val/test shrink to ~20% of raw.

Citation

• Base model: Qwen3-Embedding-0.6B
• GitHub Training GitHub-Repo-train-data
• License: Apache 2.0 (inherited from the base) thore@MacBook-Pro-von-Thore projects % cat model_card.md

license: apache-2.0 library_name: sentence-transformers base_model: Qwen/Qwen3-Embedding-0.6B datasets:

• xthor/Qwen3-Embedding-GraphQL-v1 pipeline_tag: sentence-similarity tags:
• sentence-transformers
• sentence-similarity
• feature-extraction
• graphql
• retrieval
• embeddings
• text-embeddings-inference
• qwen3
• gguf language:
• en

Qwen3-Embedding-0.6B-GraphQL

A small, self-hosted embedding model for retrieving GraphQL Type.field coordinates from natural-language questions. It is designed for schema retrieval in AI agent pipelines, where the full schema is too large to place in context for every request.

Real schemas reuse field names extensively. Many types expose fields such as title, author, createdAt, state, or status, so matching the field name alone is not enough; retrieval must also identify the correct owner type.

This model fine-tunes Qwen/Qwen3-Embedding-0.6B for that owner-type disambiguation task. At 0.6B parameters, it can run locally on CPU or alongside an agent's primary model.

Benchmark

The GraphQL AI Working Group benchmark evaluates 816 natural-language queries across GitHub, GitLab, Linear, Shopify, and the Singapore Open Data API. Field-level recall against the ground-truth Type.field coordinate:

The fine-tune leads field-level retrieval at both cutoffs and improves recall@50 by 5.1 percentage points over its base model. At the owner-type level, text-embedding-3-large leads at 87.8% recall@50; the fine-tune reaches 85.6%, ahead of the base model at 78.6% and text-embedding-3-small at 80.8%.

The model ships as SentenceTransformers weights and GGUF builds for llama.cpp and Ollama.

Important: how you format the corpus matters as much as the model. Use SDL snippets or dot_plus_gloss formatting for best results. See Embedding style comparison for details.

Inference

SentenceTransformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("xthor/Qwen3-Embedding-0.6B-GraphQL")

query = "When did I place this order?"
# coordinates of Type.field pairs
coords = [
    "Order.createdAt",
    "Order.completedAt",
    "Order.updatedAt",
]

q = model.encode(query, prompt_name="query")
c = model.encode(coords, prompt_name="document")
scores = (q @ c.T).tolist()

for coord, score in sorted(zip(coords, scores), key=lambda x: -x[1]):
    print(f"{score:.3f}  {coord}")

The fine-tuned model ranks Order.createdAt first; the un-tuned base model ranks Order.completedAt first. This is a small sanity check using readable bare coordinates. For production retrieval, use the richer SDL or gloss formatting described below.

Two prompts are wired into the model and must be used for best results:

• prompt_name="query" for natural-language questions
• prompt_name="document" for GraphQL coordinate descriptions in the corpus

Ollama

# pull one quantization (Q8_0 is a good default: near-lossless, ~650 MB)
hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./model-q8_0.gguf
EOF
ollama create qwen3-graphql-embedder -f Modelfile

# OpenAI-compatible embeddings endpoint
curl -s http://localhost:11434/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-graphql-embedder","input":"When did I place this order?"}' \
  | jq '.data[0].embedding'

llama.cpp

hf download xthor/Qwen3-Embedding-0.6B-GraphQL model-q8_0.gguf --local-dir .

./llama-server -m model-q8_0.gguf --embedding --port 8080
# POST http://localhost:8080/embedding   { "content": "..." }

Available GGUF quantizations

Results

223 held-out test queries · 28,893-coordinate corpus · 30% real SDLs (GitHub GHES, Saleor, Shopify, AniList) never seen in training.

Where the lift comes from

Direct questions ("has my package shipped?", "what's my total?") are already handled well by the base model. The gains come from indirect questions where the user names a concept rather than a field. Those require owner-type reasoning, and that's where the base model falls behind.

Small inference example

"When did I place this order?"

Against the bare coordinates Order.createdAt, Order.completedAt, and Order.updatedAt, the base model ranks Order.completedAt first. The fine-tuned model ranks the correct coordinate, Order.createdAt, first and opens a clearer margin:

Every candidate is a plausible timestamp on the same order. The model must distinguish when the order was placed from when it was completed or last updated. This three-candidate example is an illustrative sanity check; the aggregate results above measure retrieval quality over the full held-out corpus.

Known limitations

1. Formatting sensitivity. With raw dot notation (Type.field), the fine-tune's R@1 is only 0.308 on the GitHub schema. Always use sdl, dot_plus_gloss, or natural formatting for the corpus.

2. Same-owner wrong-field rate. same_owner_wrong_field_rate@1 rose from 0.063 to 0.103. The model picks the right owner type more often but occasionally lands on the wrong field within that type. The training signal rewards owner disambiguation; within-owner field disambiguation isn't targeted. The next iteration will add competition sets that share owner and differ by field.

3. Tail regression with raw dot notation. When using raw dot notation, the fine-tune's P95 rank (404) is worse than the base model's (123). The model becomes more confident: it either ranks the correct answer first or misses much harder. This is fully mitigated by using sdl (P95 40) or dot_plus_gloss (P95 41) formatting.

4. Indirect queries. Queries that don't name or allude to the owner type (e.g., "get the README" → Repository.object) remain hard for both models. The fine-tune does not improve on these.

How you format the corpus matters

How you turn each Type.field coordinate into text before embedding it affects retrieval more than the fine-tune does. The benchmark below compares twelve formats on the GitHub GraphQL schema (52 held-out queries):

Use one of these two. They tie at the top:

# sdl: if you parse the schema (MRR 0.723)
type PullRequest { baseRefName: String! }

# dot_plus_gloss: string-only, no parsing needed (MRR 0.715)
PullRequest.baseRefName — the base ref name of a pull request

The cheap string-only gloss costs almost nothing versus full schema parsing, so reach for dot_plus_gloss unless you already have parsed types on hand. Whatever you do, don't embed raw Type.field identifiers. With dot formatting, MRR drops to 0.393 and the worst-case rank blows out 10x. The owner type is what carries the signal: drop it entirely and retrieval collapses to MRR ~0.05.

Full results

Each format is one way of rendering PullRequest.baseRefName into text before embedding (the example column shows exactly what). P95 is the 95th-percentile rank, i.e. how badly the worst queries rank. Lower is better.

Training

Both: --max-seq-length 256, 4 hard negatives per anchor, bf16, full fine-tune (no LoRA), single H100. Published checkpoint: qwen3-e3.

Dataset

Built from 7,626 raw seed pairs via world-leakage, per-row strict-leakage, and family-level semantic-dedup filters. The strict-leakage filter is aggressive on real-SDL queries, which is why val/test shrink to ~20% of raw.

Citation

• Base model: Qwen3-Embedding-0.6B
• GitHub Training GitHub-Repo-train-data
• License: Apache 2.0 (inherited from the base)