BERT-based AI-Generated Academic Text Detector

A high-accuracy BERT model for detecting AI-generated academic text with 99.57% accuracy on paragraph-level samples.

Online Demo

🌐 Try the model online: https://followsci.com/ai-detection

Free web interface with real-time detection, no installation or API key required.

Model Details

Model Description

• Model Type: BERT-base-uncased fine-tuned for binary text classification
• Architecture: BERT-base-uncased (110M parameters)
• Task: Binary classification (Human-written vs AI-generated text)
• Input: Academic text paragraphs (up to 512 tokens)
• Output: Binary label (0 = Human-written, 1 = AI-generated) with confidence scores

Training Information

• Training Samples: 1,487,400 paragraph-level samples
• Validation Samples: 185,930 paragraph-level samples
• Test Samples: 185,930 paragraph-level samples
• Total Dataset: 1,859,260 paragraphs
• Training Data:
  • Human-written: Academic papers from arXiv
  • AI-generated: Text generated by various large language models (GPT, Claude, etc.)

Performance

Test Set Results

Confusion Matrix (Test Set)

Inference Speed: ~20,900 samples/second on RTX 3090 (batch size 64)

Usage

Quick Start

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load model and tokenizer
model_name = "followsci/bert-ai-text-detector"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)
model.eval()

# Detect AI text
text = "Your academic paragraph here..."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    ai_prob = probs[0][1].item() * 100
    human_prob = probs[0][0].item() * 100
    
    print(f"AI-generated probability: {ai_prob:.1f}%")
    print(f"Human-written probability: {human_prob:.1f}%")
    
    if ai_prob > 50:
        print("Prediction: AI-generated")
    else:
        print("Prediction: Human-written")

Batch Processing

texts = [
    "First paragraph...",
    "Second paragraph...",
    # ... more texts
]

inputs = tokenizer(
    texts,
    return_tensors="pt",
    truncation=True,
    max_length=512,
    padding=True
)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    
    for i, prob in enumerate(probs):
        ai_prob = prob[1].item() * 100
        print(f"Text {i+1}: AI probability = {ai_prob:.1f}%")

Using with Transformers Pipeline

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="followsci/bert-ai-text-detector",
    tokenizer="followsci/bert-ai-text-detector"
)

result = classifier("Your text here...")
print(result)

Training Details

Training Configuration

• Base Model: bert-base-uncased
• Batch Size: 64
• Learning Rate: 5e-5 (with linear warmup)
• Warmup Steps: 5,000
• Max Sequence Length: 512
• Optimizer: AdamW
• Epochs: 3
• Training Time: ~11 hours (on RTX 3090)

Dataset Distribution

Limitations

1. Domain Specificity: The model is trained primarily on academic text. Performance may degrade on:

   • Casual text or social media content
   • Technical documentation
   • Creative writing

2. Binary Classification: The model only distinguishes between "human" and "AI" text, without:

   • Identifying which AI model generated the text
   • Providing confidence intervals
   • Detecting partially AI-assisted text

3. Paragraph-Level Detection: The model is optimized for paragraph-level samples:

   • Performance on sentence-level or full-document level may vary
   • Best results achieved with structured academic paragraphs

4. False Positives: Approximately 0.82% false positive rate means some human-written text may be flagged as AI-generated.

Ethical Considerations

• Use Case: This model is intended as a tool for academic integrity and research purposes
• Bias: The model may reflect biases present in the training data
• Misuse: Should not be used as the sole criterion for academic misconduct decisions
• Transparency: Results should be interpreted with context and domain expertise

License

This model is licensed under the MIT License.

Contact

• Email: raffoduanedonnenfeld@gmail.com