RoBERTa All-Sources AI Text Identifier
This model is a roberta-base sequence classifier fine-tuned to distinguish human-written text from AI-generated text, with emphasis on creative-writing and prose-like distributions.
Labels:
human: label0ai: label1
Intended Use
The model is intended for research and internal evaluation of AI-generated prose detection. It should not be used as the sole basis for high-stakes decisions about authorship, academic integrity, employment, moderation, or user sanctions. Outputs are classifier probabilities, not proof of authorship.
Loading
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "frankhanhj/roberta-base-all-sources-ai-identifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
Training Data
The training corpus is datasets/all-sources-ai-identifier, assembled from the local sources below. Validation and test rows were not used for training. For chunked external sources, splits were isolated by group_id / original_index; the recorded split audit found 0 leaks across 10,992 checked groups.
| Source | Train Rows | Validation Rows | Test Rows | Labels / Notes |
|---|---|---|---|---|
| MAGE / HC3 / RAID mixed research corpus | 10,000 | 1,600 | 1,600 | Balanced human/AI mixture from hc3, mage, and raid |
| RL Claude creative-writing SFT outputs | 3,549 | 442 | 446 | AI-only generated writing |
| Gale Creativewriting GPT outputs | 23,986 | 2,843 | 2,960 | AI-only generated creative writing |
human-writing-dpo | 21,501 | 2,670 | 2,629 | chosen_human as human, rejected_ai as AI |
| Human creative MAGE holdout | 800 | 100 | 100 | Human-only roct_human and wp_human |
Overall split sizes:
| Split | Rows | Human | AI |
|---|---|---|---|
| Train | 59,836 | 18,322 | 41,514 |
| Validation | 7,655 | 2,473 | 5,182 |
| Test | 7,735 | 2,401 | 5,334 |
Techniques Tried
Several detector families were trained or evaluated before selecting this checkpoint:
| Run | Technique | Train Rows | Test Rows | Test Accuracy | Test ROC AUC |
|---|---|---|---|---|---|
roberta-base-all-sources-ai-identifier | RoBERTa sequence classifier on all sources | 59,836 | 7,735 | 0.984874 | 0.998999 |
roberta-base-mixed-plus-claude-ai-identifier | RoBERTa sequence classifier on mixed research + Claude | 16,000 | 1,600 | 0.991250 | 0.999239 |
roberta-base-mixed-research-ai-identifier | RoBERTa sequence classifier on MAGE/HC3/RAID | 10,000 | 1,600 | 0.989375 | 0.999022 |
token-qwen35-mixed-research-ai-identifier | Qwen3.5 token-level CNN/attention classifier | 10,000 | 1,600 | 0.981875 | 0.998786 |
qwen35-2b-mixed-research-embedding-identifier | Pooled Qwen3.5 embeddings + MLP | 10,000 | 1,600 | 0.971875 | 0.998188 |
raw-mixed-research-ai-identifier | Raw text n-gram + stylometric classifier | 10,000 | 1,600 | 0.965625 | 0.989867 |
raw-mixed-plus-claude-ai-identifier | Raw text n-gram + stylometric classifier on mixed + Claude | 16,000 | 1,600 | 0.950000 | 0.985572 |
qwen35-2b-mage-embedding-identifier | Pooled Qwen3.5 embeddings + MLP on MAGE | 40,000 | 6,000 | 0.861500 | 0.939538 |
roberta-base-mage-4k-ai-identifier | RoBERTa sequence classifier on small MAGE subset | 4,000 | 1,000 | 0.738000 | 0.883386 |
token-qwen35-mage-4k-ai-identifier | Qwen3.5 token-level CNN/attention classifier on small MAGE subset | 4,000 | 1,000 | 0.772000 | 0.864189 |
raw-mage-ai-identifier | Raw text n-gram + stylometric classifier on MAGE | 40,000 | 6,000 | 0.730000 | 0.813246 |
The selected model is the all-sources RoBERTa checkpoint. It has slightly lower original mixed-benchmark accuracy than the earlier mixed-plus-Claude RoBERTa run, but it fixes a severe false-positive failure on human-writing-dpo.
Training Procedure
Base model: roberta-base
Key settings:
- Max sequence length:
512 - Batch size:
8 - Evaluation batch size:
64 - Gradient accumulation:
2 - Epochs:
2 - Learning rate:
2e-5 - Weight decay:
0.01 - Warmup ratio:
0.06 - Class weighting: balanced
- Mixed precision:
fp16 - Seed:
20260530 - Best checkpoint selection: validation ROC AUC
The selected checkpoint is epoch 1. Epoch 2 increased AI recall slightly but reduced human recall, so epoch 1 was kept.
Evaluation
All-sources held-out test result:
| Accuracy | ROC AUC | Human Recall | AI Recall | Confusion Matrix |
|---|---|---|---|---|
| 0.984874 | 0.998999 | 0.967097 | 0.992876 | [[2322, 79], [38, 5296]] |
Comparison against the previous mixed-plus-Claude RoBERTa model on the all-sources test set:
| Model | Accuracy | ROC AUC | Human Recall | AI Recall |
|---|---|---|---|---|
| all-sources RoBERTa | 0.984874 | 0.998999 | 0.967097 | 0.992876 |
| previous mixed-plus-Claude RoBERTa | 0.792889 | 0.706478 | 0.433153 | 0.954818 |
Important held-out slice results:
| Test Slice | Accuracy | Human Recall | AI Recall |
|---|---|---|---|
human-writing-dpo | 0.977178 | 0.960027 | 1.000000 |
| Gale Creativewriting GPT | 0.989527 | n/a | 0.989527 |
| RL Claude Creative Writing SFT | 1.000000 | n/a | 1.000000 |
| Human creative MAGE | 0.940000 | 0.940000 | n/a |
| MAGE mixed | 0.973333 | 0.960000 | 0.986667 |
| HC3 | 0.997500 | 0.995000 | 1.000000 |
| RAID | 0.995000 | 1.000000 | 0.990000 |
Known Limitations
- The model is sensitive to the distributions represented in the assembled datasets.
- AI text detectors can produce false positives, especially under domain shift.
- The training set is not balanced by source; balanced class loss was used, but source imbalance remains.
- Some evaluation slices are single-label, so they report only human recall or AI recall rather than full binary metrics.
- The model should be calibrated and re-evaluated before use on new domains, languages, or generator families.
Artifacts
This repository contains the full fine-tuned Transformers checkpoint:
model.safetensorsconfig.jsontokenizer.jsontokenizer_config.jsonmetrics.jsonhistory.jsonrun_config.jsongroup_metrics.json