Model Details
This model is designed to classify customer service inquiries into four customer priorites: Disruption but can work, Disruption cannot work, Disruption several cannot work and Disruption department cannot work.
Training Data
The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt
Training Procedure
The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.
Model Performance
The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:
-
Confusion Matrix Analysis
- Disruption but can work: The model correctly classified 800 instances. However, it misclassified 200 instances as "Disruption cannot work," 21 as "Disruption several cannot work," and 48 as "Disruption department cannot work."
- Disruption cannot work: There are 440 correct predictions in this category, with 270 instances incorrectly classified as "Disruption but can work," 32 as "Disruption several cannot work," and 43 as "Disruption department cannot work."
- Disruption several cannot work: Only 99 instances are correctly identified within this category, with misclassifications including 69 instances as "Disruption but can work," 46 as "Disruption cannot work," and 16 as "Disruption department cannot work."
- Disruption department cannot work: This category has 100 correctly classified instances, with 95 misclassified as "Disruption but can work" and 86 as "Disruption cannot work."
-
Learning Curve Analysis
- Training Loss: There is a consistent decrease in training loss from approximately 2.0 to 1.4, indicating the model's increasing accuracy on the training dataset.
- Validation Loss: The validation loss starts just above 1.0 and ends slightly below this value, showing minimal change and suggesting the model's learning has plateaued on the validation dataset.
-
Interprating Model's Output:
- LABEL_0 stands for Disruption but can work
- LABEL_1 stands for Disruption cannot work
- LABEL_2 stands for Disruption department cannot work
- LABEL_3 stands for Very Disruption several cannot work