ScamShield — Scam & Phishing Detection Model
A fine-tuned DistilBERT model that classifies text as scam or safe — built to run fully on-device (browser/edge), as the detection engine behind the ScamShield Chrome extension.
Model Description
This model detects scam, phishing, and fraudulent job/internship messages from plain text — SMS messages, emails, job offer letters, and similar written content. It was fine-tuned from distilbert-base-uncased for binary sequence classification (0 = safe, 1 = scam).
It's designed to be small and fast enough to run entirely client-side (in a browser via ONNX + Transformers.js, or on edge devices), so no user text needs to be sent to a server for scam detection.
- Base model:
distilbert-base-uncased - Task: Binary text classification (safe vs. scam)
- Language: English
- License: MIT
Intended Use
- Detecting phishing/scam SMS and emails
- Detecting fraudulent job/internship offers (fake recruiters, upfront-fee scams)
- Powering privacy-preserving, on-device scam-detection tools (browser extensions, edge apps)
Not intended for: legal/compliance decisions, moderating content at scale without human review, or as a sole determinant of fraud — see Limitations below.
Training Data
Combined from three sources:
- SMS Spam Collection Dataset — real SMS messages, ham/spam labeled
- EMSCAD — Employment Scam Aegean Dataset — real vs. fraudulent job postings
- Custom synthetic contrastive dataset (~255 examples, generated via LLM) — pairs of professionally-worded job offers that are identical in structure and language, differing only in whether an upfront payment is requested. This was added after testing revealed the model initially missed softly-worded scams that avoided obvious keywords (see "Model Iteration History" below).
Training Procedure
- Fine-tuned for 2 epochs, learning rate 2e-5, batch size 16, weighted cross-entropy loss (to address class imbalance — scam examples are a minority class)
- Trained on a single RTX 4050 (6GB VRAM) with fp16 mixed precision
- Max sequence length: 512 tokens
Evaluation Results (held-out test set)
| Metric | Score |
|---|---|
| Accuracy | 98.5% |
| Precision | 0.944 |
| Recall | 0.833 |
| F1 | 0.885 |
Model Iteration History
This model went through 4 iterations, which is worth documenting honestly since it shapes how the model should be used:
- v1: Baseline on the two public datasets. Confidently correct on obvious scams, but confidently wrong (near 0%) on professionally-worded scams avoiding obvious red-flag keywords.
- v2: Added broad synthetic augmentation to fix this — overcorrected, started flagging legitimate offer letters as scams due to generic HR language ("verification," "documentation").
- v3: Rebalanced with more safe examples — overcorrected the other way, lost recall on real scams.
- v4 (this model): Used a contrastive dataset — scam/safe pairs with identical structure and language, differing only in whether payment was requested. This isolated the actual decisive signal and fixed both prior failure modes.
Limitations
- Trained and evaluated primarily on job/internship offers and SMS-style messages — may not generalize well to other scam formats (e.g. crypto scams, romance scams) without further fine-tuning.
- English only.
- Can still be uncertain on genuinely ambiguous, real-world borderline messages — in the ScamShield extension, this model's output is combined with a rule-based red-flag layer as a safety net, rather than used alone. Using this model standalone without such a safety net is not recommended for high-stakes use.
- Like any small classifier, it can be sensitive to phrasing changes near its decision boundary.
How to Use
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("rehan-ml/scamshield-scam-detector")
model = AutoModelForSequenceClassification.from_pretrained("rehan-ml/scamshield-scam-detector")
text = "Congratulations! You've been selected. Pay a refundable registration fee of $50 to confirm your position."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
outputs = model(**inputs)
scam_probability = torch.softmax(outputs.logits, dim=1)[0][1].item()
print(f"Scam probability: {scam_probability:.4f}")
Related
- 🔗 ScamShield Chrome Extension (GitHub) — the full project this model powers, including the browser extension, rule-based safety net, and training notebook.
Author
Built by Rehan Raza for OSDHack 2026.