Intent Detection V2.4 (V12.10)
Health-app user intent classifier + safety router fine-tuned from Qwen2.5-7B-Instruct. Routes user messages into 5 states: PASS, REDFLAG, BLOCK, STEER, or COEXIST (emergency + personal-dose).
Performance (val set: 2518 samples)
| Metric | Score |
|---|---|
| Safety F1 (REDFLAG ∪ BLOCK) | 92.3% |
| Multi-label EM | 88.0% |
| Joint acc (intent + all types) | 81.5% |
| Single-label acc (v11-compatible projection) | 90.1% |
| Intent acc | 93.5% |
| Coexist recall (REDFLAG + BLOCK) | 76.2% |
| Parse err | 0.16% (of which ~half are eval-side bugs — model output is valid) |
| REDFLAG reason_code acc | 98.3% |
Improvements vs V11 (single-label):
- Safety F1: 90.9% → 92.3% (+1.4pp)
- REDFLAG F1: 90.0% → 94.1% (+4.1pp)
- New multi-label capability: 76.2% coexist recall (v11 was 0%)
Output Schema (V12.10 flat)
6 fixed keys, deterministic order:
{
"intent": "HA|HCN|MT|PI|OTHER",
"redflag": true|false,
"redflag_code": "RF-1_GENERAL|RF-2_PEDIATRIC|RF-3_OBSTETRIC|RF-4_ELDERLY|RF-5_IMMUNOCOMP|\"\"",
"block": "BLOCK_HARD|STEER_SOFT|PASS|NONE",
"block_code": "OOS_H_*|OOS_S_*|PASS_*|\"\"",
"entities": {
"drugs": ["<verbatim from user text>"],
"diseases": ["..."],
"symptoms": ["..."]
}
}
Entities are verbatim substrings of the user message (no translation/normalization).
5 Valid State Combinations
| State | redflag | block | Use case |
|---|---|---|---|
| PASS | false | PASS | Normal health knowledge Q |
| REDFLAG only | true | NONE | Emergency without personal-dose Q |
| BLOCK only | false | BLOCK_HARD | Personal action / off-topic |
| STEER only | false | STEER_SOFT | Personal clinical judgment |
| COEXIST | true | BLOCK_HARD | Emergency + personal-dose |
Training data
- 54,404 samples (English + Chinese + Bruneian Malay)
- Base 27K + STEER×3 + coexist×8 + patch13×3 (targeted narrow rules) + patch16×5 (anti-drift + narrow)
- 57.1% samples have non-empty entities (GPT-4o back-filled)
Usage with vLLM
python -m vllm.entrypoints.openai.api_server \
--model ningpy/intent-detection-V2.4 \
--served-model-name intent \
--dtype float16 \
--port 8001
Client sample
See https://git.evyd.tech/peiyan.ning/intent-model-v12/tree/main/sample for a Python client with routing + response templates.
Recommended request settings:
temperature: 0.0max_tokens: 200stop: [](rely on model's natural<|im_end|>)
Downstream routing
| Model output | Action |
|---|---|
redflag=true, block="BLOCK_HARD" | Emergency guidance + refuse dose advice |
redflag=true, block="NONE" | Emergency guidance |
redflag=false, block="BLOCK_HARD" | Refuse + refer to doctor |
redflag=false, block="STEER_SOFT" | Soft guide to consultation |
redflag=false, block="PASS" | Normal RAG/LLM answer |