Multilingual AI-vs-Human Text Detector — XLM-RoBERTa-base

A binary classifier that detects whether a passage was written by a human or generated by an AI model, fine-tuned from XLM-RoBERTa-base to work across English, Chinese, and Vietnamese with a single checkpoint.

Built as an AI content forensics research project: the full system — data pipeline, statistical baselines, per-language evaluation, REST API, and web demo — lives in the companion repository.

• 📦 GitHub (full project): https://github.com/vutuongvy101/multilingual-ai-human-text-detection
• 👩‍💻 Author: Tuong Vy Vu — GitHub · LinkedIn

Why a multilingual detector?

Monolingual detectors fail badly outside their pre-training language: in the same experimental setup, an English-only DistilBERT drops to F1 0.784 on Chinese (vs 0.977 on English) because Chinese characters tokenise into meaningless subword fragments. This model uses multilingual pre-training to hold F1 0.946–0.989 across all three languages — evidence that cross-lingual AI-content detection requires multilingual representations, not per-language models.

Evaluation

Test-set F1 (70/15/15 prompt-level split, seed 42; 90 test samples per language, 270 total):

Comparison against baselines trained on the same data (full table, confusion matrices, and analysis in the GitHub README):

Note that the TF-IDF logistic regression baseline is competitive in-domain — the value of this transformer model is expected in robustness to paraphrase and vocabulary shift, which n-gram features cannot capture (see Limitations).

Training

• Base model: FacebookAI/xlm-roberta-base (~279M params), sequence classification head, F32.
• Data: 900 QA pairs (300 per language), each with one human and one AI answer:
  • English — HC3 (reddit_eli5)
  • Chinese — HC3-Chinese (open_qa)
  • Vietnamese — crawled from Vietnamese Reddit communities
  • AI answers generated by Qwen2.5-1.5B-Instruct
• Setup: fine-tuned 3 epochs, learning rate 2e-5, max_length=256, prompt-level 70/15/15 train/val/test split (seed 42) so no question appears in both train and test.
• Reproduce: python scripts/train_transformer.py --model-name xlm-roberta-base in the GitHub repo.

Usage

from transformers import pipeline

detector = pipeline(
    "text-classification",
    model="bibbbu/multilingual-ai-human-detector_xlm-roberta-base",
)

texts = [
    "Honestly I just left it overnight and it worked fine, no idea why lol",
    "There are several important factors to consider when addressing this question.",
]
print(detector(texts))
# [{'label': 'human', 'score': ...}, {'label': 'ai', 'score': ...}]

For batch inference, a FastAPI server, and a Streamlit demo, see the GitHub repository.

Intended use & limitations

Intended use: research on cross-lingual AI-text detection; educational and portfolio use; experimentation with content-authenticity pipelines.

Limitations — read before relying on predictions:

• ⚠️ Do not use this model alone for high-stakes decisions (academic-misconduct accusations, content moderation enforcement, hiring). AI-text detectors produce false positives, and non-native writers are a known false-positive risk for detectors in general.
• Generator specificity: AI-labelled training text comes from a single generator (Qwen2.5-1.5B-Instruct). Detection of text from other models (GPT-4-class, Claude, Gemini) is untested and likely weaker.
• Domain shift: trained on QA-style forum answers; performance on news, legal, academic, or social-media text is untested.
• Small evaluation set: 270 test samples total; strong scores should be confirmed on larger, held-out domains before any production use.
• Language coverage: English, Chinese, Vietnamese only.

Citation

@misc{vu2024multilingual,
  title  = {Multilingual AI-Human Text Detection},
  author = {Vu, Tuong Vy},
  year   = {2024},
  url    = {https://github.com/vutuongvy101/multilingual-ai-human-text-detection}
}