ATC-Pilot Speaker Role Classification Model
This is a binary sequence classification model designed to determine whether a given air traffic communication utterance originates from a pilot or an air traffic controller (ATC), based on text alone.
Traditionally, speaker role attribution in air traffic communication relies on acoustic features such as voice characteristics and channel separation. This model departs from that convention by tackling the task entirely in the text domain, using a transformer-based architecture fine-tuned for speaker role prediction.
Task Description
The model performs binary classification on single-turn utterances to assign one of two speaker roles:
PILOTATC
It is fine-tuned using a DeBERTa-v3-large model on manually processed and labeled air traffic communication transcripts.
Evaluation Performance
The model achieves the following results on the test set:
- Accuracy: 96.64%
- Precision: 96.40%
- Recall: 96.91%
- F1 Score: 96.65%
Preprocessing & Training Setup
A custom preprocessing pipeline was used to prepare the training data, including:
- Speaker attribution heuristics based on known call sign and phrase patterns
- Phrase normalization
- Text standardization
- Filtering of irrelevant utterances
- Dataset balancing
Each utterance is treated independently and labeled for speaker role classification.
Model Architecture
- Base model:
microsoft/deberta-v3-large - Task type:
SequenceClassification(num_labels=2) - Training setup:
- Trained on 2x H100 80GB SXM5
- Cosine learning rate schedule with warmup (10%)
- Batch size: 128
- Early stopping based on F1 score
- Max sequence length: 256 tokens
- Mixed-precision training (FP16)
- Evaluation every 200 steps
Intended Use
This model is designed for:
- Speaker role tagging in ATC communication transcripts
- Preprocessing for multi-modal ATC systems
- Filtering or structuring large corpora of aviation text for downstream tasks
Limitations
- Operates on single-turn utterances only; no turn-level or dialogue context is used
- Ambiguous transmissions like "ROGER" or "THANK YOU" may be difficult to classify using text alone
- Additional modalities (e.g., audio features, metadata) may be required for full disambiguation
Example Predictions
Input: "CLEARED FOR TAKEOFF RUNWAY ONE ONE LEFT"
Prediction: "ATC"
Input: "REQUESTING PUSHBACK"
Prediction: "PILOT"
Benchmark Comparison
This model improves upon prior transformer-based models for text-only speaker role classification. For comparison, a related model by Juan Zuluaga-Gomez, based on BERT-base, achieved the following:
- Accuracy: 89.03%
- Precision: 87.10%
- Recall: 91.63%
- F1 Score: 89.31%
The fine-tuned DeBERTa-v3-large model presented here significantly outperforms this baseline:
- Accuracy: 96.64%
- Precision: 96.40%
- Recall: 96.91%
- F1 Score: 96.65%
Jupyter notebooks are included to reproduce and compare evaluations:
evaluate_juans_model.ipynbevaluate_jacks_model.ipynb
These evaluate both models using the same test set and print detailed classification metrics.