Scam/Spam Message Detector — India (DistilBERT, fine-tuned)
Fine-tuned version of distilbert-base-uncased-finetuned-sst-2-english
for binary classification of scam/spam vs. legitimate ("ham") text messages, with a focus on Indian scam patterns
(lottery fraud, Aadhaar/KYC phishing, UPI/bank fraud, fake job offers, OTP-theft attempts).
Labels
0/ham: Legitimate message1/spam: Scam or spam message
Training Data
- Base dataset:
scam_hum_india.csv, real-world Indian SMS/message samples (telecom promos, government notices, casual messages), deduplicated. - Augmented with ~200 additional synthetic examples covering underrepresented scam categories: lottery/prize fraud, Aadhaar/KYC phishing, UPI/bank fraud, fake job offers, and OTP-theft social engineering — added after identifying that the base dataset was skewed toward telecom promotional spam and missed these patterns.
- Class-weighted loss (
sklearncompute_class_weight="balanced") used during training to counter class imbalance (~1521 ham vs ~700 spam originally).
Training Procedure
- Base model:
distilbert-base-uncased-finetuned-sst-2-english - Epochs: 3
- Learning rate: 2e-5
- Batch size: 16
- Weighted
CrossEntropyLossvia customTrainersubclass to address class imbalance
Evaluation Results
| Metric | Score |
|---|---|
| Accuracy | 0.9971 |
| F1 | 0.9963 |
| Eval Loss | 0.0206 |
Limitations
- Trained primarily on India-specific scam message patterns (Aadhaar, UPI, telecom, lottery in Indian Rupees); may not generalize well to other regions or currencies.
- English-tokenizer based — code-mixed Hindi/English (Hinglish) text may reduce accuracy.
- Subtle, low-signal scam messages (no amounts, brand names, or urgency cues) are harder to detect and may be misclassified as legitimate.
- A portion of spam examples are synthetically generated to patch category gaps; performance on real-world messages of these types should be validated further before production use.
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="anmolshrivastav/distilbert-scam-detector-india")
clf("Congratulations! You've won 50000 rupees, click link to claim")
Quantized ONNX Version
A dynamically quantized (INT8) ONNX version model.onnx is available in the onnx/ subfolder for faster CPU inference with a smaller footprint. Also a onnx/model_fp32.onnx (unquantized ONNX version)
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline
model = ORTModelForSequenceClassification.from_pretrained(
"anmolshrivastav/distilbert-scam-detector-india", subfolder="onnx"
)
tokenizer = AutoTokenizer.from_pretrained(
"anmolshrivastav/distilbert-scam-detector-india", subfolder="onnx"
)
clf = pipeline("text-classification", model=model, tokenizer=tokenizer)
clf("Congratulations! You've won 50000 rupees, click link to claim")