PropertyPilot Triage Classifier
DistilBERT finetuned on 10,980 synthetic property maintenance tickets with two classification heads:
- Category (10 classes): Appliances, Common Areas, Electrical, Elevator, HVAC, Noise, Pests, Plumbing, Security, Structural
- Urgency (4 classes): P1 Emergency (4h), P2 Urgent (24h), P3 Standard (3-5d), P4 Scheduled (7-14d)
Part of the PropertyPilot project — an AI-powered property maintenance triage system.
Performance
| Task | Zero-shot F1 | Finetuned F1 | Improvement |
|---|---|---|---|
| Category (macro) | 0.032 | 0.972 | +0.939 |
| Urgency (macro) | 0.062 | 0.761 | +0.700 |
Urgency labels are derived from ticket text content (keyword-based rule classifier), making them strongly correlated with the actual maintenance scenario described.
Dataset
Trained on propertypilot/property-pilot-tickets:
- 13,726 clean rows, 10 categories, 4 urgency tiers
- Brackets/template placeholders stripped, ispers corruption fixed
- 80/10/10 stratified train/val/test split
Model Architecture
DistilBertModel (distilbert-base-uncased)
-> CLS token -> Dropout(0.1)
-> Linear(768, 10) # category head
-> Linear(768, 4) # urgency head
Training
- Epochs: 5, LR: 2e-5, Batch: 32, Max seq len: 128
- AdamW + linear warmup (10%) + gradient clipping
- Joint loss: CrossEntropy(category) + CrossEntropy(urgency)
Files
| File | Description |
|---|---|
model.pt | PyTorch state dict (best checkpoint by combined val F1) |
label_map.json | id2label / label2id for both heads |
tokenizer_config.json | Tokenizer config (distilbert-base-uncased fast tokenizer) |
tokenizer.json | Fast tokenizer file |
distilbert_config.json | Base DistilBERT config |
eval_results.json | Full training history + test metrics |
confusion_cat.png | Category confusion matrix |
confusion_urg.png | Urgency confusion matrix |
Inference
import torch, json
from transformers import DistilBertModel, DistilBertTokenizerFast
import torch.nn as nn
class TriageClassifier(nn.Module):
def __init__(self, n_cat, n_urg):
super().__init__()
self.bert = DistilBertModel.from_pretrained("distilbert-base-uncased")
self.dropout = nn.Dropout(0.1)
self.cat_head = nn.Linear(768, n_cat)
self.urg_head = nn.Linear(768, n_urg)
def forward(self, input_ids, attention_mask):
out = self.bert(input_ids=input_ids, attention_mask=attention_mask)
cls = self.dropout(out.last_hidden_state[:, 0])
return self.cat_head(cls), self.urg_head(cls)
# Load
label_map = json.load(open("label_map.json"))
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")
model = TriageClassifier(n_cat=10, n_urg=4)
model.load_state_dict(torch.load("model.pt", map_location="cpu"))
model.eval()
# Predict
text = "gas leak in kitchen, can smell it strongly"
enc = tokenizer(text, return_tensors="pt", max_length=128, truncation=True, padding="max_length")
with torch.no_grad():
cat_logits, urg_logits = model(enc["input_ids"], enc["attention_mask"])
cat = label_map["category_id2label"][str(cat_logits.argmax(-1).item())]
urg = label_map["urgency_id2label"][str(urg_logits.argmax(-1).item())]
print(f"Category: {cat} | Urgency: {urg}")