SES Industry Email Classifier (BERT Japanese)
🎯 A BERT-based model for automatically classifying business emails in Japan's SES (System Engineering Service) industry
This model classifies emails commonly exchanged in Japan's IT staffing industry with high accuracy, enabling automated email routing and workflow optimization.
📊 Performance
| Metric | Score |
|---|---|
| Accuracy | 99.51% |
| F1 (weighted) | 0.99 |
| Precision (weighted) | 1.00 |
| Recall (weighted) | 1.00 |
Per-Class Performance
| Class | Precision | Recall | F1-Score | Support |
|---|---|---|---|---|
| 案件 (Project) | 1.00 | 1.00 | 1.00 | 486 |
| 要員 (Talent) | 0.99 | 1.00 | 1.00 | 721 |
| その他 (Other) | 1.00 | 0.60 | 0.75 | 15 |
📌 Model Overview
| Item | Description |
|---|---|
| Base Model | tohoku-nlp/bert-base-japanese-v3 |
| Task | 3-class classification |
| Language | Japanese |
| Max Input | 512 tokens (~1000 characters) |
| Training Data | ~4,900 SES industry emails |
| Test Data | ~1,200 emails (held-out) |
🏷️ Classification Labels
| ID | Label | Description | Example |
|---|---|---|---|
| 0 | 案件 (Project) | Project/job opportunity postings | "【案件】PM募集 60万〜 新宿 即日" |
| 1 | 要員 (Talent) | Engineer/talent introduction | "【ご紹介】インフラエンジニア 30代" |
| 2 | その他 (Other) | Other business emails | Meeting requests, general correspondence |
🚀 Usage
Pipeline (Recommended)
from transformers import pipeline
classifier = pipeline(
"text-classification",
model="naoki-hosokawa/ses-mail-classifier-bert-japanese"
)
# Project email example
result = classifier("【案件】Java開発 60万〜80万 渋谷 即日〜長期 面談1回")
print(result)
# [{'label': '案件', 'score': 0.98}]
# Talent email example
result = classifier("【ご紹介】Javaエンジニア 40代男性 都内在住 即日稼働可能 希望単価55万")
print(result)
# [{'label': '要員', 'score': 0.95}]
Direct Model Usage
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
model_name = "naoki-hosokawa/ses-mail-classifier-bert-japanese"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
text = "【案件】Python開発 リモート可 50万〜"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
outputs = model(**inputs)
predictions = torch.softmax(outputs.logits, dim=-1)
labels = ["案件", "要員", "その他"]
predicted_label = labels[predictions.argmax().item()]
confidence = predictions.max().item()
print(f"Classification: {predicted_label} (Confidence: {confidence:.2%})")
Batch Processing
from transformers import pipeline
classifier = pipeline(
"text-classification",
model="naoki-hosokawa/ses-mail-classifier-bert-japanese",
device=0 # Use GPU
)
emails = [
"【案件】AWS構築 フルリモート 55万〜",
"【ご紹介】クラウドエンジニア 経験5年",
"明日の会議は15時からでお願いします",
]
results = classifier(emails, batch_size=32)
for email, result in zip(emails, results):
print(f"{result['label']}: {email[:30]}...")
🎓 Training Details
Training Data
- Source: Real business emails from Japan's SES industry
- Preprocessing: Subject + body concatenated, truncated to 1000 characters
- Split: 4,399 train / 489 validation / 1,222 test
Training Configuration
- Epochs: 5
- Batch Size: 8
- Learning Rate: 5e-5 (default)
- Warmup Steps: 10
- Weight Decay: 0.01
- Best Model Selection: Based on F1 score
Training Environment
- Google Colab (T4 GPU)
- Training Time: ~38 minutes
⚠️ Limitations
- Domain-Specific: Optimized for SES industry emails; may not generalize to other email types
- "Other" Class: Lower recall (60%) due to limited samples; consider LLM fallback for low-confidence predictions
- Input Length: Texts exceeding 512 tokens will be truncated
- Language: Japanese only
🔧 Technical Details
Why BERT?
| Aspect | BERT | LLM (GPT, etc.) |
|---|---|---|
| Inference Speed | ◎ Fast | △ Slow |
| Inference Cost | ◎ Low | × High |
| Classification Accuracy | ○ Good | ◎ Excellent |
| Batch Processing | ◎ Efficient | △ Costly |
For high-volume email processing (100K+ emails/month), using BERT as a pre-filter significantly reduces LLM API costs while maintaining accuracy.
📜 License
Apache License 2.0
This model follows the license of the base model tohoku-nlp/bert-base-japanese-v3.
🙏 Acknowledgments
- Tohoku NLP Lab - Japanese BERT model
- Hugging Face - Transformers library
📬 Author
- Author: Naoki Hosokawa
- Organization: Apple Seed LLC (合同会社アップルシード)
- Location: K-1 Building 3F, 1-8-18 Ebisu, Shibuya-ku, Tokyo, Japan
⭐ If you find this model useful, please give it a star!