distil-qwen3-1.7b-posthog-prioritizer
A 1.7B specialist that ranks a batch of deduplicated product findings by impact, frequency, and effort, returning strict JSON with a reason per item. One of three tools in the distil-posthog-traffic-analyser harness; trained on the Distil Labs platform.
Task contract
Input:
Findings (N):
[ {"id": "...", "kind": "bug|gap", "severity": 1-5,
"occurrences": <int>, "title": "...", "evidence": "..."}, ... ]
Return JSON now. Every id above must appear in "ranked" exactly once.
Output — valid JSON only, matching:
{"ranked":[{"id":"<input id>","rank":1,"reason":"<=2 sentences citing the numbers"}]}
Hard constraint: every input id appears exactly once — no dropped, invented, or duplicated ids. Ranking judgment encoded in training: security issues lead regardless of frequency; revenue blockers rank above equally severe non-revenue issues; silent data corruption outranks loud errors; at equal severity, reach wins; cosmetic items rank last.
Training
- Base model: Qwen3-1.7B (Apache 2.0)
- Teacher: openai.gpt-oss-120b (Apache 2.0)
- Seed data: 22 hand-authored, schema-validated ranking batches
(single-item to 4-item, including equal-severity ties and
security/revenue trade-offs), committed at
examples/seeds/prioritizer.jsonl(18 train / 4 held-out test); a coverage validator guarantees every seed batch ranks each finding exactly once - Synthetic expansion: 10,004 examples generated and validated by the Distil Labs platform from the seed set
- Method: platform-managed fine-tune (task type: question-answering, JSON output)
A 0.6B variant was trained first; it ranked held-out batches correctly but violated the exactly-once coverage constraint on a batch containing near-duplicate findings. The 1.7B student holds exact coverage on that same adversarial batch, which is why this size ships.
Evaluation
Held-out test set (n=4), scored by the platform's LLM judge (an answer fails outright on any coverage violation, making this judge deliberately strict):
| Untrained Qwen3-1.7B | Teacher (gpt-oss-120b) | This model | |
|---|---|---|---|
| LLM-as-a-Judge | 75.00% (3/4) | 75.00% (3/4) | 75.00% (3/4) |
| ROUGE | 38.96% | 47.17% | 50.31% |
The judge score saturates because both pass its format bar on most batches; live behavior separates them. This model, on live batches (this GGUF via Ollama, matching the hosted endpoint): exact id coverage on 6/6 — 4 held-out seed batches, one adversarial near-duplicate batch, and one live pipeline batch fed by the other two tools — with rank-1 choices matching the hand-written gold ranking on 3 of the 4 held-out batches and a defensible judgment call on the fourth.
Usage (Ollama)
ollama create posthog-prioritizer -f Modelfile # FROM ./<this gguf>
Then in the harness .env:
TOOL_PRIORITIZER_MODEL=posthog-prioritizer