distil-qwen3-1.7b-posthog-extractor
A 1.7B specialist that reads a short user-behavior narration and emits product findings (bugs and UX gaps) as strict JSON. One of three tools in the distil-posthog-traffic-analyser harness; trained on the Distil Labs platform.
Task contract
Input:
Narration:
"""
<3-sentence session narration>
"""
Return JSON now.
Output — valid JSON only, matching:
{"findings":[{"kind":"bug|gap","severity":1-5,"title":"...","evidence":"..."}]}
bug= something visibly broke or errored;gap= something the user clearly wanted to do but couldn'tseverity: 1 (cosmetic) to 5 (blocks a core flow)- A clean session correctly yields
{"findings":[]} - No markdown, no commentary, nothing invented beyond the narration
Training
- Base model: Qwen3-1.7B (Apache 2.0)
- Teacher: openai.gpt-oss-120b (Apache 2.0)
- Seed data: 25 hand-authored, schema-validated narration/findings pairs
— including 6 clean-session examples with empty findings — committed at
examples/seeds/extractor.jsonl(20 train / 5 held-out test) - Synthetic expansion: 10,078 examples generated and validated by the Distil Labs platform from the seed set
- Method: platform-managed fine-tune (task type: question-answering, JSON output)
A 0.6B variant was trained first and scored similarly on the platform judge, but live testing showed it missed low-signal findings (dead-end searches with no error keyword) entirely; the 1.7B student catches them, which is why this size ships.
Evaluation
Held-out test set (n=5), scored by the platform's LLM judge:
| Untrained Qwen3-1.7B | Teacher (gpt-oss-120b) | This model | |
|---|---|---|---|
| LLM-as-a-Judge | 40.00% (2/5) | 80.00% (4/5) | 80.00% (4/5) |
| ROUGE | 60.25% | 67.61% | 68.88% |
The tuned 1.7B student matches its 120B teacher's judge score on this task.
Live checks on this GGUF via Ollama match the hosted endpoint: 8/10 on an adversarial suite (held-out narrations plus narrations authored by the narrator SLM). The two misses are bug-vs-gap label confusion on ambiguous "silent failure" cases — the finding is still surfaced with a sensible title and severity; only the kind tag wobbles.
Head-to-head vs a frontier teacher (bun run eval, gpt-5-mini as opponent
and judge, 5 demo narrations): 3 ties, 1 win for this model, 1 win for the
teacher — with this model running locally at $0 per call.
Usage (Ollama)
ollama create posthog-extractor -f Modelfile # FROM ./<this gguf>
Then in the harness .env:
TOOL_EXTRACTOR_MODEL=posthog-extractor