llm-d-sc-cost
A 384-dimensional sentence embedding model fine-tuned to separate expected serving cost, for use as a routing signal by llm-d-sc.
The model does not emit a class directly. It produces an embedding that is ranked against labelled
anchors (anchors.json, shipped here), so the taxonomy stays replaceable data rather than a
frozen classification head.
Taxonomy
| Tier | Meaning |
|---|---|
MINIMAL | A single fact or one-step instruction; trivial output |
LOW | A short answer: a brief explanation, a one-liner, a small snippet |
MODERATE | One substantial deliverable: a complete guide, a full module, a detailed comparison |
HIGH | Very large input or output: analysing a whole corpus, or producing an exhaustive document |
Cost is related to complexity but is not the same signal. Summarising a long document is expensive and simple; a short proof is cheap and hard. This taxonomy tracks the size of the work, not its difficulty.
Intended use
Budgeting and admission decisions: which requests justify a large context window, a long generation budget, or a more expensive endpoint. The classifier emits ranked evidence only; the routing and budgeting decisions remain the caller's.
Evaluation
Evaluated by llm-d-sc on a held-out set of 60 prompts authored independently of the training corpus, 15 per tier, of which 20 are deliberate boundary cases. Method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.
| Model | Accuracy | Macro F1 | Boundary cases |
|---|---|---|---|
| llm-d-sc-cost (this model) | 0.8333 | 0.8342 | 0.7500 |
all-MiniLM-L6-v2 (base, same anchors) | 0.7500 | 0.7459 | 0.6000 |
Confusion (rows are truth, columns predicted):
| MINIMAL | LOW | MODERATE | HIGH | |
|---|---|---|---|---|
| MINIMAL | 13 | 2 | 0 | 0 |
| LOW | 1 | 11 | 3 | 0 |
| MODERATE | 0 | 0 | 13 | 2 |
| HIGH | 0 | 0 | 2 | 13 |
Every error is an ADJACENT-tier confusion. For an ordinal taxonomy that is the benign failure mode: a request misread by one tier gets a slightly wrong budget, not a wrong destination. There are no MINIMAL-to-HIGH style errors.
Latency on CPU (single thread, embed plus rank): p50 8.5 ms, p99 10.1 ms.
Produced on one contributor's homelab; not independently reproduced.
Training
Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and
group_by_label batch sampling, 12 epochs, batch size 64, lr 2e-5.
Data: 2650 synthetic prompts generated by Qwen3.8-27B across 40 domains using real-world, boundary and hard-negative strategies, then re-labelled independently by DeepSeek-V4-Flash. Rows where the two models disagreed were dropped, which removed 34% of the corpus, leaving 1416 balanced examples (354 per tier). A 34% disagreement rate is itself informative: cost tiers are genuinely more ambiguous than complexity tiers, and the dropped rows were mostly MODERATE/HIGH boundary cases.
The held-out evaluation set was authored by hand before the training data existed and is asserted to share no prompt with it.
Limitations
- English only.
- Trained on synthetic data; no human-labelled validation set exists.
- Cost depends on your serving stack. These tiers describe the size of the requested work, not the price of any particular deployment.
- Anchor quality directly determines accuracy. Replacing
anchors.jsonchanges behaviour without retraining.
License
Apache-2.0.