DistilBERT Base Uncased - Domain Adaptation, Fine-Tuned on IMDB Movie Reviews (Standard Subword Masking)
This model is a domain-adapted version of distilbert-base-uncased fine-tuned on the
IMDB movie reviews dataset for Masked Language Modeling (MLM).
By adapting the base model to informal, domain-specific film terminology, its language modeling performance improved significantly over the base checkpoint.
Model Description
- Base Model:
distilbert-base-uncased - Task: Domain Adaptation / Masked Language Modeling (Fill-Mask)
- Dataset:
stanfordnlp/imdb(all splits concatenated and chunked into 128-token blocks) - Objective: Adapt general language representations to informal, domain-specific movie review language and vocabulary co-occurrences.
Performance & Results
- Evaluation Metric: Cross-Entropy Loss & Perplexity
- Pre-training Perplexity: 23.14
- Fine-tuned Perplexity: 9.91
Note: For Masked Language Modeling, the loss evaluated during training is the cross-entropy over masked tokens. A standard metric for measuring domain adaptation is Perplexity, calculated as
exp(eval_loss). Lower perplexity indicates that the model is less "surprised" by text from the target domain.
Training Details
- Training Dataset Size: 50,000 sequence chunks (128 max token length, full packing / zero padding)
- Evaluation Dataset Size: 10,000 sequence chunks (Official IMDB test split)
- Hardware: NVIDIA T4 GPU (Google Colab)
- Collator Strategy: Standard Subword Masking (
DataCollatorForLanguageModeling, 15% probability for dynamic masking)
Note: A Whole-Word Masking (WWM) Collator variant is available in
distilbert-base-uncased-finetuned-imdb-wwm, together with its results and a discussion of the comparison.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- weight_decay: 0.01
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 3.0
- precision: FP16 mixed-precision training (fp16=True, native PyTorch AMP)
Training results
| Training Loss | Epoch | Step | Validation Loss | Model Preparation Time |
|---|---|---|---|---|
| 2.4848 | 1.0 | 782 | 2.3654 | 0.0046 |
| 2.4338 | 2.0 | 1564 | 2.3231 | 0.0046 |
| 2.4176 | 3.0 | 2346 | 2.2983 | 0.0046 |
Framework versions
- Transformers 5.13.1
- Pytorch 2.11.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
Usage
from transformers import pipeline
fill_mask = pipeline("fill-mask", model="AlexStamp/distilbert-base-uncased-finetuned-imdb-standard")
# Example domain completion
result = fill_mask("This movie was completely [MASK] and worth every penny!")
for prediction in result:
print(f"{prediction['token_str']}: {prediction['score']:.4f}")