SIFTA — Multilingual Fake News Detector (EN + UR)
Model: Fine-tuned xlm-roberta-base for binary fake news classification
Languages: English · Urdu (اردو)
Labels: TRUE (0) · FAKE (1)
Student: Muhammad Umair Khan · 2023F-BCET-010
Institute: Sir Syed University of Engineering and Technology, Karachi
Project: SIFTA — AI Misinformation Detection Platform (Final Year Project 2025–26)
Model Description
This model fine-tunes XLM-RoBERTa base on a combined English + Urdu fake news corpus. It is the core ML component of SIFTA — a multi-modal AI misinformation detection platform that also performs voice clone detection, document forgery analysis, and live source verification.
The raw probability output is passed through SIFTA's Decision Engine alongside live source verification signals to produce a five-class Trust Score verdict: Verified Real · Likely Real · Uncertain · Unable to Verify · Likely Fake.
Training Data
| Dataset | Samples (train) | Language |
|---|---|---|
| ISOT | 32482 | English |
| UrduFakeNews-AxToGrind | 8054 | Urdu |
| Total | 40536 | EN + UR |
Dataset details
Urdu — Ax-to-Grind UrduFakeNews Corpus
- 10,106 news items from Pakistani social media and news outlets
- Binary labels: TRUE / FAKE
- Near-balanced: ~5,039 FAKE / ~4,997 TRUE
- Average length: 351 characters per item
LIAR (Wang, 2017)
- 12,836 political statements from PolitiFact
- 6-way labels mapped to binary:
- pants-fire / false / barely-true → FAKE
- half-true / mostly-true / true → TRUE
ISOT Fake News Dataset (University of Victoria)
- ~44,000 real/fake news articles
- Real: Reuters. Fake: unreliable sources. Already binary labelled.
Training Configuration
| Parameter | Value |
|---|---|
| Base model | xlm-roberta-base |
| Max token length | 256 |
| Epochs | 5 |
| Batch size | 16 x 2 grad accum = 32 effective |
| Learning rate | 2e-05 |
| LR schedule | Cosine with 10% linear warmup |
| Weight decay | 0.01 |
| Dropout | 0.1 |
| Loss | CrossEntropyLoss with inverse-frequency class weights |
| Early stopping | Patience 3 on val macro-F1 |
| Mixed precision | fp16 on T4 GPU |
| Random seed | 42 |
Evaluation Results
Overall (held-out test set — 10% stratified)
| Metric | Score |
|---|---|
| Accuracy | 0.9696 |
| Macro F1 | 0.9695 |
| TRUE (class 0) F1 | 0.9676 |
| FAKE (class 1) F1 | 0.9713 |
Per-language macro-F1
| Language | Macro F1 |
|---|---|
| EN | 0.9854 |
| UR | 0.9053 |
Full classification report
precision recall f1-score support
TRUE 0.9632 0.9721 0.9676 2369
FAKE 0.9753 0.9674 0.9713 2698
accuracy 0.9696 5067
macro avg 0.9693 0.9698 0.9695 5067
weighted avg 0.9697 0.9696 0.9696 5067
Confusion Matrix
| Predicted TRUE | Predicted FAKE | |
|---|---|---|
| Actual TRUE | 2,303 (TN) | 66 (FP) |
| Actual FAKE | 88 (FN) | 2,610 (TP) |
- TRUE recall: 97.2%
- FAKE recall: 96.7%
Usage
HuggingFace Inference API (used by SIFTA backend)
import httpx
API_URL = "https://api-inference.huggingface.co/models/zodumair/sifta-fakenews-xlmr"
headers = {"Authorization": "Bearer hf_your_token_here"}
def classify(text):
return httpx.post(API_URL, headers=headers, json={'inputs': text}).json()
print(classify("Scientists confirm new vaccine is 77% effective."))
print(classify("حکومت نے خفیہ طور پر ملک کا سارا سونا بیرون ملک بھیج دیا"))
Local inference
from transformers import pipeline
classifier = pipeline("text-classification", model="zodumair/sifta-fakenews-xlmr")
print(classifier("SHOCKING: Scientists hiding cancer cure from the public."))
# [{'label': 'FAKE', 'score': 0.93}]
Limitations
- Optimised for headlines and short excerpts under 256 tokens
- Urdu performance is lower than English due to smaller Urdu training set
- Satire and opinion pieces may be misclassified
- Use alongside source verification — not as a standalone decision tool
Citation
@misc{sifta2026,
author = {Muhammad Umair Khan},
title = {SIFTA: AI Misinformation Detection Platform},
year = {2026},
institution = {Sir Syed University of Engineering and Technology, Karachi},
note = {Final Year Project, Batch 2023F}
}
SIFTA FYP SSUET Karachi · Batch 2023F · 2023F-BCET-010