Model Details
This model is designed to classify customer service inquiries into five categories: Technical Issues, Billing & Payment, Product Inquiries, Account Management, and Policy Questions.
Training Data
The model was trained on a balanced dataset of 53000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt
Training Procedure
The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.
Model Performance
The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:
-
Confusion Matrix Analysis
- Technical Issues: Some misclassification noted, particularly with Policy Questions.
- Billing & Payment: High accuracy with some instances misclassified as Technical Issues.
- Product Inquiries: Very high accuracy with few misclassifications.
- Account Management: Notable confusion with Policy Questions.
- Policy Questions: Highest accuracy among all categories.
-
Learning Curve Analysis
- The training loss decreased consistently, indicating good model learning.
- The validation loss remained close to the training loss and did not show signs of increasing, suggesting that the model was generalizing well without overfitting.
-
Interprating Model's Output:
- LABEL_0 stands for Account Management
- LABEL_1 stands for Billing & Payment
- LABEL_2 stands for Policy Questions
- LABEL_3 stands for Product Inquiries
- LABEL_4 stands for Technical Issues