Model Details
This model is designed to classify customer service inquiries into two requst type: Inicident and Service Request.
Training Data
The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt
Training Procedure
The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.
Model Performance
The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:
-
Confusion Matrix Analysis
- Incident: The model has a high number of correct classifications with 2500 true positives. There are 25 false negatives where "Incident" was incorrectly classified as "Service Request".
- Service Request: There are 440 true positives where "Service Request" was correctly classified. However, there are 150 false negatives where "Service Request" was incorrectly classified as "Incident".
-
Learning Curve Analysis
- Training Loss: The training loss starts at approximately 0.20 and shows a steady decrease, ending near 0.12. This indicates that the model is effectively learning and improving its performance on the training dataset.
- Validation Loss: The validation loss begins around 0.18 and, after a slight decrease, starts to plateau around 0.16. This suggests that while the model is learning and generalizing, improvements on the validation set are limited after the initial epochs.
-
Interprating Model's Output:
- LABEL_0 stands for Incident
- LABEL_1 stands for Service Request