llm-d-sc-complexity
A 384-dimensional sentence embedding model fine-tuned to separate prompt complexity tiers, for use as a routing signal by llm-d-sc, the semantic classification runtime for llm-d.
The model does not emit a class directly. It produces an embedding that is ranked against a set of
labelled anchors (anchors.json, shipped with this repository). This keeps the taxonomy as data
rather than as a frozen classification head: anchors can be replaced or extended without retraining.
Taxonomy
| Tier | Meaning |
|---|---|
SIMPLE | Single-fact lookup or a one-step instruction |
MEDIUM | One substantive task: a function, an explanation with an example, a short guide |
COMPLEX | Multi-component design or build with several interacting concerns |
REASONING | Proof, derivation, or formal analysis |
Intended use
Selecting a serving tier per request. A SIMPLE prompt does not need a frontier model; a REASONING
prompt usually does. The classifier emits ranked evidence only. Routing, endpoint selection, and
session affinity remain the caller's responsibility.
Evaluation
Evaluated by llm-d-sc on a held-out set of 80 prompts authored independently of the training corpus, 20 per tier, deliberately drawn from domains outside the training pipeline's domain-transfer list (sailing, agriculture, transit, broadcast, museums, orchestras). 20 of the 80 are boundary cases.
Classification method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.
| Model | Accuracy | Macro F1 | Boundary cases |
|---|---|---|---|
| llm-d-sc-complexity (this model) | 0.9750 | 0.9749 | 0.9500 |
all-MiniLM-L6-v2 (base, same anchors) | 0.6250 | 0.6234 | 0.6500 |
Per tier:
| Tier | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| SIMPLE | 1.000 | 1.000 | 1.000 | 20 |
| MEDIUM | 0.909 | 1.000 | 0.952 | 20 |
| COMPLEX | 1.000 | 0.900 | 0.947 | 20 |
| REASONING | 1.000 | 1.000 | 1.000 | 20 |
Both errors are COMPLEX predicted as MEDIUM, the same boundary the training-time report
identified as the model's only remaining confusion. The base model's confidence is near-uniform
(0.25-0.27 across four tiers), which is the expected signature of an embedding space that carries
no complexity structure at all.
Latency on CPU (single thread, Apple M-series, embed plus rank): p50 9.2 ms, p99 19.0 ms.
These numbers were produced on a homelab and have not been independently reproduced.
Training
Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and
group_by_label batch sampling on 871 synthetic examples generated and cross-verified by two
separate LLMs. Pipeline: https://github.com/cnuland/hello-chris-sr-finetuned
Usage
from sentence_transformers import SentenceTransformer
import json, numpy as np
model = SentenceTransformer("cnuland/llm-d-sc-complexity")
anchors = json.load(open("anchors.json"))["anchors"]
def classify(text, top_k=3):
q = model.encode(text, normalize_embeddings=True)
scores = {}
for tier, examples in anchors.items():
sims = model.encode(examples, normalize_embeddings=True) @ q
scores[tier] = float(np.sort(sims)[-top_k:].mean())
return max(scores, key=scores.get), scores
print(classify("Prove that the square root of 3 is irrational."))
# ('REASONING', {...})
Limitations
- English only.
- Trained on synthetic data; no human-labelled validation set exists.
- The
MEDIUM/COMPLEXboundary is genuinely ambiguous and is where residual error concentrates. - Anchor quality directly determines accuracy. Replacing
anchors.jsonchanges behaviour without retraining.
License
Apache-2.0.