Telco_Transformer_V1
The objective of the 'Telco Transformer' initiative is to pre-train a language model for the telecom industry to understand complex, contextual relationships in domain specific text data. Business hypothesis hinges on non-standard natural languages, such as components of a telecom system, technical terminology, and rich knowledge from multiple subdomains. This builds a strong case for pretraining a model from scratch. It constitutes a custom tokenizer to capture telco vocabulary, large scale unsupervised pre-training that is paired with supervised fine tuning to perform well on downstream tasks. This model will be able to complete sentences, and answer questions with accuracy that is superior to RAG or a fine tuned model.
Model V1 achieves the following results on the evaluation set:
- Loss: 3.8171
Input Token Examples
RAN, IP Network, Radio, 5G, Core Network
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- distributed_type: sagemaker_data_parallel
- num_devices: 8
- total_train_batch_size: 128
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 300
- num_epochs: 30
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 5.3308 | 1.57 | 500 | 5.2874 |
| 4.9378 | 3.14 | 1000 | 4.8655 |
| 4.5929 | 4.72 | 1500 | 4.5568 |
| 4.4308 | 6.29 | 2000 | 4.3593 |
| 4.2703 | 7.86 | 2500 | 4.2217 |
| 4.1977 | 9.43 | 3000 | 4.1222 |
| 4.0986 | 11.01 | 3500 | 4.0477 |
| 4.0791 | 12.58 | 4000 | 3.9904 |
| 3.9625 | 14.15 | 4500 | 3.9470 |
| 3.9381 | 15.72 | 5000 | 3.9114 |
| 3.9399 | 17.3 | 5500 | 3.8844 |
| 3.9146 | 18.87 | 6000 | 3.8640 |
| 3.8779 | 20.44 | 6500 | 3.8468 |
| 3.844 | 22.01 | 7000 | 3.8355 |
| 3.8364 | 23.58 | 7500 | 3.8266 |
| 3.8566 | 25.16 | 8000 | 3.8216 |
| 3.8411 | 26.73 | 8500 | 3.8187 |
| 3.815 | 28.3 | 9000 | 3.8172 |
| 3.8225 | 29.87 | 9500 | 3.8171 |
Framework versions
- Transformers 4.28.1
- Pytorch 2.0.0
- Datasets 2.16.1
- Tokenizers 0.13.3