Qwen2.5-0.5B Intent Router (LoRA fine-tuned)
A 500M-parameter sequence classifier fine-tuned to route user queries to one of four tool categories for an internal GenAI assistant, achieving 95.3% accuracy and 3.3x lower latency than a 70B-parameter LLM baseline on the same task.
Model Description
This model classifies a user query into exactly one of four categories, intended to sit at the entry point of an AI agent and decide which downstream tool/node should handle the request:
| Category | Description |
|---|---|
rag_tool | Query needs internal company knowledge (HR policy, internal docs, onboarding, internal tools) |
code_tool | Query needs code written, debugged, explained, or reviewed |
web_search | Query needs current/live/recent external information |
direct_answer | Query is answerable from general knowledge, no lookup needed |
Base model: Qwen/Qwen2.5-0.5B-Instruct Fine-tuning method: LoRA (r=8, alpha=16, dropout=0.1) applied to attention projections (q_proj, k_proj, v_proj, o_proj), then merged into the base weights for standalone deployment. Task head: Sequence classification (AutoModelForSequenceClassification), not generative/instruction-following output, chosen specifically for single-forward-pass inference speed and zero output-parsing risk in production.
Intended Use
Designed as the routing node in a multi-tool AI agent (e.g. a LangGraph graph), where a fast, cheap classification decision is needed before dispatching to a more expensive tool (RAG pipeline, code execution, web search, or direct LLM response). Not intended as a general-purpose chat model -- it has no generative/conversational capability; it only outputs one of the four labels.
Training Data
581 synthetically generated employee queries (Llama-3.3-70B via Groq), covering 4 balanced categories, with two deliberate design choices:
- ~20% of examples per category were generated as near-boundary/ambiguous cases (e.g., a query mentioning a coding library but actually asking for current release info), so the benchmark reflects realistic routing difficulty rather than only trivially separable examples.
- Targeted augmentation after error analysis: an initial training run (530 examples) showed the model confusing technical-vocabulary queries with code_tool even when the query was actually asking for current/live info (e.g. "has there been an update to the AWS CLI recently"). 60 additional web_search examples specifically pairing technical vocabulary with recency framing were generated and added, improving web_search recall from 84.4% to 90.6%, with a minor, expected trade-off on two adjacent code_tool boundary cases (a normal small-data fine-tuning dynamic, not a regression -- net test accuracy was unchanged at 95.3%).
Data quality controls:
- Exact-hash deduplication during generation
- TF-IDF cosine-similarity near-duplicate detection and removal, run separately for (a) train-vs-test/val leakage and (b) train-internal redundancy
- Test set generated in a separate batch/prompt pass from train, specifically to avoid the test set being near-paraphrases of training examples
Train/val/test split: 581 / 94 / 127 examples. Test set was evaluated exactly once, after training was finalized, to ensure a fair held-out estimate.
Evaluation Results
Accuracy comparison (vs. zero-shot LLM baseline)
Benchmarked against an independent LLM (zero-shot prompted, no examples, same held-out 127-query test set) as a fair, methodologically-clean comparison point -- the LLM judge was not used to generate the training data, to avoid contaminating the comparison.
| Metric | This model (0.5B, fine-tuned) | LLM baseline (zero-shot) |
|---|---|---|
| Accuracy | 95.3% | 96.9% |
| Macro F1 | 95.2% | 96.6% |
Latency comparison
Measured as real wall-clock time per query: local GPU forward pass (this model) vs. a real API call to a 70B LLM via Groq (chosen specifically because Groq is among the fastest LLM inference providers available -- making this a conservative, harder-to-dispute speed comparison rather than picking an easy target).
| Metric | This model | LLM baseline (Groq, 70B) |
|---|---|---|
| Mean latency | 110 ms | 365 ms |
| Speedup | 3.3x faster | -- |
| Parameter count | 0.5B | 70B (140x larger) |
Per-class breakdown (this model, final test run)
| Category | Precision | Recall | F1 |
|---|---|---|---|
| code_tool | 0.941 | 0.941 | 0.941 |
| direct_answer | 0.926 | 0.962 | 0.943 |
| rag_tool | 0.946 | 1.000 | 0.972 |
| web_search | 1.000 | 0.906 | 0.951 |
Honest Limitations
- The LLM baseline scores slightly higher on raw accuracy (96.9% vs 95.3%). This model's value proposition is not "beats the LLM outright" -- it is near-parity accuracy at 140x fewer parameters and 3.3x lower latency, which is the relevant trade-off for a high-throughput routing node that runs on every single user message.
- Two of the four LLM-baseline misclassifications, and several of this model's misclassifications, cluster on the same two genuinely ambiguous test examples (queries that blend web_search framing with general-knowledge framing), suggesting these specific examples may sit at a real boundary in the category definitions themselves, not purely a model weakness.
- Trained on synthetic, LLM-generated data -- phrasing diversity, while deliberately varied, reflects the generating model's style and may not fully capture real employee phrasing patterns in a live deployment.
- Dataset size (581 train examples) is small; performance on categories or phrasing patterns not represented in training is untested.
Training Procedure
- Hardware: Google Colab T4 GPU
- LoRA: r=8, alpha=16, dropout=0.1, targeting q_proj/k_proj/v_proj/o_proj
- Trainable parameters: 1,084,928 / 495,121,280 (0.22%)
- Epochs: 8, learning rate 2e-4, batch size 16
- Tracked via MLflow
Citation
If you use this model, please reference the base model:
Qwen2.5 Technical Report, Qwen Team, Alibaba Group