BERT Email Phishing Detector
Fine-tuned bert-base-uncased for
binary classification of email/message text as phishing or legitimate.
Labels
| id | label |
|---|---|
| 0 | legitimate |
| 1 | phishing |
Intended use
- Research and education on phishing detection.
- A component / baseline in a larger email-security pipeline.
Not intended as a standalone production email security gateway. Always keep a human in the loop for high-stakes decisions, and combine with URL reputation, sender authentication (SPF/DKIM/DMARC) and other signals.
How to use
from transformers import pipeline
clf = pipeline("text-classification", model="<your-username>/bert-phishing-detector")
clf("Verify your account now at http://secure-login.example.ru")
# [{'label': 'phishing', 'score': 0.99}]
Training
- Base model:
bert-base-uncased - Objective: binary sequence classification (cross-entropy)
- Max sequence length: 256 tokens
- Optimizer / LR: AdamW, 2e-5, weight decay 0.01, 6% warmup
- Epochs: 3 (best checkpoint by validation F1)
Training code: src/train.py in the project repository.
Evaluation
Fine-tuned for 3 epochs on ~65.7k emails and evaluated on an 8,207-email held-out test split:
| Metric | Score |
|---|---|
| Accuracy | 0.9948 |
| Precision | 0.9956 |
| Recall | 0.9944 |
| F1 | 0.9950 |
These scores are very high partly because the training data (the Kaggle phishing-email collection) is cleanly separable. Expect lower performance on noisier, real-world email — validate on your own data before relying on it.
Limitations & biases
- Distribution shift is the big one. The training corpus is heavily preprocessed (lowercased, punctuation and stop-words removed) and the legitimate emails skew toward early-2000s corporate mail (Enron). The model scores ~99% on held-out data from this same distribution, but can misclassify raw, modern, or very short emails (e.g. a casual "Hi team, notes attached" message may be flagged as phishing). For best results, feed it text preprocessed the same way as training, and treat the headline metrics as in-distribution — not a promise of real-world accuracy.
- Trained primarily on English text; performance on other languages is untested.
- Phishing tactics evolve quickly; the model can go stale and should be periodically retrained on fresh data.
- May produce false positives on legitimate marketing emails that share surface features with phishing (urgency, links, "verify" language).
- The bundled
data/sample_emails.csvis a tiny illustrative sample. Train on a large, representative corpus before relying on any metric.
Data
Bring your own labeled corpus with a text column and an integer label
column (0 = legitimate, 1 = phishing). Commonly used public sources include the
Nazario phishing corpus, the Enron email dataset (legitimate mail), SpamAssassin
public corpus, and various Kaggle phishing-email datasets. Check each dataset's
license before redistribution.
Ethical considerations
This model classifies text and can make mistakes. Do not use it to automatically delete mail or take irreversible actions without human review.