bert-base-ud-ewt-pos
Fine-tuning of bert-base-uncased for Part-of-Speech (POS) tagging, delivered as
part of assignment U2T01 (Adapting BERT for NLP tasks — Trends in Data Science,
Unit 2, Universidad Politécnica de Yucatán).
Model description
bert-base-uncased with a token-level linear classification head over the last
hidden state of every token, predicting one of 17 Universal POS (UPOS) tags (NOUN,
VERB, ADJ, DET, PUNCT, etc.).
Training data
- Dataset: UD English EWT
(
universal-dependencies/universal_dependencies, configen_ewt). - Note: the dataset ID given in the original assignment prompt
(
universaldependencies/universal_dependencies) has a typo; the correct organization name on the Hub uses a hyphen (universal-dependencies). - Same per-token label shape as NER, so the same subword-alignment convention is
used: the label goes on the first subword of each word,
-100everywhere else.
Training procedure
Two adaptation methods were trained and compared; full fine-tuning is the delivered model.
| Hyperparameter | Value |
|---|---|
| Base model | bert-base-uncased (110M params) |
| Method | Full fine-tuning (BERT body + head, jointly) |
| Learning rate (head) | 1e-3 |
| Learning rate (BERT body) | 2e-5 |
| Epochs | 4 |
| Batch size | 32 (train) / 64 (eval) |
| Seed | 42 |
| Trainable parameters | ~108,900,000 |
| Training time | 6.3 min (single T4 GPU) |
Compared alternative (not delivered): partial fine-tuning, freezing the entire BERT body except its last 2 encoder layers (~14,190,000 trainable params, 2.1 min).
Evaluation results
Accuracy is the primary metric (POS has no multi-token "entity" concept, so per-token accuracy is directly meaningful); macro F1 is reported as a secondary metric so rare tags (INTJ, SYM, X) are not hidden behind frequent ones (NOUN, PUNCT).
| Method | Test Accuracy | Test Macro F1 |
|---|---|---|
| Partial fine-tuning (last 2 layers + head) | 95.59% | 89.19% |
| Full fine-tuning (delivered) | 97.62% | 93.81% |
The gap between methods depends heavily on the metric: only 2.03 points in accuracy, but 4.62 points in macro F1 — partial fine-tuning already handles frequent tags well but falls behind specifically on rare ones.
Intended uses & limitations
- Intended use: part-of-speech tagging of English text following the Universal Dependencies UPOS tagset, for coursework/research.
- Limitations: trained only on UD English EWT (web text domain); performance on rare tags (INTJ, SYM, X) is weaker than on frequent ones, and this may be more pronounced on out-of-domain text. Single seed, 4 epochs, no extensive hyperparameter search.
References
- Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
- Hugging Face. Fine-tune a pretrained model. https://huggingface.co/docs/transformers/training
- Dataset: universal-dependencies/universal_dependencies
(config
en_ewt)