Model Details
This model is designed to classify customer service inquiries into nine services: SAP ERP, Atlassian, Adobe, Salesforce, Reporting, Microsoft Power Platform, Microsoft SharePoint, Snowflake, Microsoft Office.
Training Data
The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt
Training Procedure
The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.
Model Performance
The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:
-
Confusion Matrix Analysis
- SAP ERP: Most instances are classified correctly (400), with a few misclassifications as Adobe (1) and Microsoft Office (1).
- Atlassian: This category has perfect classification with all instances (410) correctly identified.
- Adobe: Also has high accuracy with 390 instances correctly classified and a single misclassification as SAP ERP (1).
- Salesforce: There are 450 correctly classified instances, but there's some confusion with Reporting (16) and Microsoft Power Platform (1), and Microsoft SharePoint (1).
- Reporting: There are 59 correct predictions. However, a significant number of Reporting instances are misclassified as Salesforce (36).
- Microsoft Power Platform: This category has 320 correct classifications, with a few instances misclassified as Reporting (5), Snowflake (2), and Microsoft SharePoint (2).
- Microsoft SharePoint: Most instances are correctly classified (390), with minimal confusion with other services.
- Snowflake: There are 300 instances correctly identified, with a single instance misclassified as Microsoft Power Platform (1).
- Microsoft Office: This category has 30 instances, all correctly classified with no misclassifications.
-
Learning Curve Analysis
- Training Loss: The training loss starts at approximately 0.12 and decreases to about 0.065. The steady decline indicates the model is learning effectively from the training data.
- Validation Loss: The validation loss starts around 0.11, decreases, then increases slightly at epoch 2 before declining again, ending around 0.075. This pattern suggests some variation in model performance on the validation set, but overall, the validation loss follows a downward trend, indicating improving model generalization.
-
Interprating Model's Output:
- LABEL_0 stands for Adobe
- LABEL_1 stands for Atlassian
- LABEL_2 stands for Microsoft Office
- LABEL_3 stands for Microsoft Power Platform
- LABEL_4 stands for Microsoft SharePoint
- LABEL_5 stands for Reporting
- LABEL_6 stands for SAP ERP
- LABEL_7 stands for Salesforce
- LABEL_8 stands for Snowflake