Model Summary
Qwen3-0.6B-Insight-Extractor is a LoRA fine-tune of Qwen/Qwen3-0.6B. It converts a raw customer-support ticket into a single structured JSON object across 9 fields, for automated ticket classification and downstream entity-extraction workflows.
| Base model | Qwen/Qwen3-0.6B |
| Fine-tuning method | LoRA Adapters via Unsloth + TRL SFTTrainer |
| Task | Structured JSON extraction from support-ticket text |
| Training data | AbhishekG711/processed_support_tickets (train split) |
| Target repo | AbhishekG711/Qwen3-0.6B-Insight-Extractor |
Base Model Architecture (Qwen3-0.6B)
| Type | Dense, causal decoder-only Transformer |
| Parameters | 0.6B total (~0.44B non-embedding) |
| Layers | 28 |
| Attention | Grouped-Query Attention — 16 query heads / 8 KV heads |
| Native context length | 32,768 tokens |
| Reasoning modes | Hybrid — supports both "thinking" (<think>...</think>) and "non-thinking" modes via enable_thinking |
| Base model license | Apache 2.0 |
Intended Use
Given a raw support ticket, produce one JSON object for automated triage, routing, sentiment/urgency dashboards, or downstream alerting — not a conversational assistant.
Fine-Tuning Training Details
| Hyperparameter | Value |
|---|---|
| Quantization | None (load_in_4bit=False) |
LoRA rank (r) | 32 |
| LoRA alpha | 64 (= 2 × r) |
| LoRA dropout | 0.0 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| LoRA bias | none |
| Max sequence length | 4096 |
| Per-device train batch size | 4 |
| Target global batch size | 32 |
| Gradient accumulation steps | 8 (= 32 // 4) |
| Epochs | 1 |
| Learning rate | 2e-4 |
| LR scheduler | linear |
| Warmup steps | 10 |
| Weight decay | 0.01 |
| Optimizer | adamw_8bit |
| Precision | bf16 if supported, else fp16 (auto-detected via is_bf16_supported()) |
| Completion-only loss | True — loss computed only on completion tokens, not the prompt |
| Sequence packing | False |
| Eval strategy | every 25 steps |
| Save strategy | every 50 steps, save_total_limit=3 |
| Load best model at end | True (metric_for_best_model="eval_loss", greater_is_better=False) |
| Gradient checkpointing | "unsloth" mode |
| Seed | 3407 |
| Experiment tracker | report_to="none" — console/log file only |
Evaluation
| Metric | Value |
|---|---|
Validation loss (eval_loss, best checkpoint) | 0.233829 |
How to Prompt This Model
The model was fine-tuned against exactly this system/user structure — matching it as closely as possible at inference time will give the best results.
System prompt:
# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.
User message template:
Analyze the following customer support email and extract structured insights.
<raw ticket text>
Expected output: One JSON object with exactly these 9 keys, in this order: is_actionable, summary, sentiment, category, intent, aspect, urgency, reported_cause, entities.
Inference example
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AbhishekG711/Qwen3-0.6B-Insight-Extractor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
SYSTEM_PROMPT = """
# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.
"""
USER_INSTRUCTION = "Analyze the following customer support email and extract structured insights.\n\n"
ticket_text = "My order #4471 arrived damaged and support hasn't replied in 3 days."
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_INSTRUCTION + ticket_text},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
enable_thinking=False,
return_tensors="pt",
).to(model.device)
output = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))