modernbert-ai-text-detector-raid-mix
RAID-specialist variant: trained with attack exposure (MAGE + stratified RAID mixture).
Fine-tuned ModernBERT-base classifier for detecting machine-generated English text (label 1 = machine). Built in the open research project robust-ai-text-detection, where every number regenerates from committed artifacts and the full research log, literature review, and negative results live.
Warning. This variant is heavily specialized to RAID-style attacked text. On the truly out-of-distribution M4GT set its TPR at 1% FPR is 0.00 — it assigns high machine-confidence to unseen-domain human text. Prefer the general checkpoint unless your inputs resemble RAID.
Metrics
Thresholds must be calibrated on human text from your own distribution; the scores below use per-dataset calibration. AUROC alone is misleading for this task — use the low-FPR columns.
| eval | AUROC | TPR@5%FPR | TPR@1%FPR |
|---|---|---|---|
| MAGE test | 0.983 | 0.933 | 0.848 |
| HC3 (cross-dataset) | 0.995 | 0.974 | 0.953 |
| RAID eval grid, incl. attacks (semi-in-distribution) | 0.992 | 0.955 | 0.904 |
| M4GT (cross-dataset) | 0.855 | 0.612 | 0.000 |
| Frontier Qwen3-2025 probe (RAID-format) | 0.999 | 0.996 | 0.985 |
Training
MAGE + stratified RAID train-pool mixture (371K rows, every generator/domain/attack cell), 1 epoch, lr 3e-5, 512 tokens.
Intended use and limitations
Research use. Output is probabilistic evidence, never proof of authorship: at 1% FPR, an institution processing 75,000 documents a year would wrongly flag ~750. Do not use as the basis of disciplinary action. Untested on non-native-writer false-positive rates; English only; document-level only; defeated by adaptive paraphrase attacks. Pair with Unicode/NFKC input normalization (see the repository) — it neutralizes homoglyph and zero-width attacks for free.
Citation
Please cite the datasets and methods this builds on (MAGE, RAID, ModernBERT — full BibTeX in the repository).