distilbert-jd-classifier
Fine-tuned distilbert-base-uncased that classifies a job description into one of four engineering tracks:
ai_mldevops_srehardware_embeddedfullstack_swe
Built to solve a real problem: given a job posting, automatically pick which resume variant to send.
Results (held-out test set, n=40)
| class | precision | recall | f1 |
|---|---|---|---|
| ai_ml | 0.91 | 1.00 | 0.95 |
| devops_sre | 0.80 | 0.80 | 0.80 |
| hardware_embedded | 1.00 | 1.00 | 1.00 |
| fullstack_swe | 0.89 | 0.80 | 0.84 |
| macro avg | 0.90 | 0.90 | 0.90 |
A note on the data quality lesson this project produced
The training set (400 rows, pulled from a Kaggle job-postings dataset plus an
HF full-text search over a 33k-row job dataset for FPGA/VLSI/embedded roles)
initially trained to 0.75 macro-F1. An audit of the worst misclassifications
surfaced 3 mislabeled rows — staffing-agency "Account Manager" / "Recruiter"
postings that got swept into hardware_embedded because they mentioned
FPGA/embedded as one of many industries the agency covers, even though the
job itself has nothing to do with engineering.
Removing those 3 rows (0.75% of the data) took macro-F1 from 0.75 → 0.90.
Code for the audit step: scripts/audit_labels.py in the
GitHub repo.
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="srijavuppala22/distilbert-jd-classifier", top_k=None)
clf(full_job_description_text, truncation=True, max_length=512)
# [{'label': 'ai_ml', 'score': 0.83}, {'label': 'devops_sre', 'score': 0.09}, ...]
Pass the full job description text, not a one-line summary — see Limitations below.
Training details
- Base model:
distilbert-base-uncased - 3 epochs, lr=2e-5, batch size 16, max_length=512
- Selection metric: macro-F1 (not accuracy — classes are imbalanced enough that accuracy is misleading)
- Trained locally on an Apple M3 (MPS backend), ~2 min/run
Limitations
- Training data is 100% public/generic postings (Kaggle + HF), not real job-search data — expect some accuracy loss on postings with unusual phrasing or hybrid roles (e.g. "MLOps Engineer" genuinely straddles
ai_mlanddevops_sre). - 4-class only; anything outside these tracks (sales, PM, data analyst, etc.) will be forced into the nearest of the 4.
- Trained on full-length JD text (hundreds of words); confidence drops sharply on short one-line inputs. A one-sentence paraphrase of a job (e.g. "hiring an ML engineer for RAG pipelines") produces near-uniform, low-confidence scores across all 4 classes because it's out of distribution from what the model saw during training. Always pass the complete posting text.
- On the 40-example held-out test set, macro-F1 measured between 0.85 and 0.90 across repeated eval runs (pipeline tokenization defaults vs. the explicit
max_length=512truncation used during training account for the small swing) — treat 0.90 as the training-run number, not a guaranteed number for every inference path.
Repo
Full pipeline (data prep, weak labeling, audit, training script, Gradio demo): github.com/srijavuppala/jd-classifier-distilbert