ResumeAI — Resume / Job-Description Matcher (MPNet, fine-tuned + calibrated)
Fine-tuned all-mpnet-base-v2 scoring resume-to-job-description fit on a 0-1 scale.
Trained with a combined CoSENT + CosineSimilarity objective on 815 curated pairs from
255 real job postings with 3x augmentation, then calibrated with Platt
scaling fitted on an external calibration split.
Evaluation
All numbers below are on 106 pairs from 53 job postings with zero overlap with training — verified, not assumed. The calibrator was fitted on a separate 106-pair external split, so the test pairs are untouched by any stage of fitting.
| Metric | Value | 95% CI |
|---|---|---|
| Spearman (production seed 43) | 0.8163 | [0.7396, 0.8638] |
| MAE | 0.1270 | [0.108, 0.1462] |
| Spearman across 3 seeds | 0.8273 +/- 0.0236 | |
| MAE across 3 seeds | 0.1194 +/- 0.0113 | |
| Base model before fine-tuning | 0.6246 / 0.2138 | |
| Precision@1 over 53 unseen postings | 84.9% | vs 25% random |
Calibrator choice: Platt and isotonic are statistically tied on this test set (bootstrap 95% CI on the MAE difference: [-0.0037, 0.014]). Platt ships because a 2-parameter sigmoid cannot overfit a 106-pair calibration split, while an isotonic step function can.
Usage
from sentence_transformers import SentenceTransformer
import numpy as np, pickle
model = SentenceTransformer("dlepighe1/resume-jd-matcher-mpnet")
emb = model.encode([resume_text, jd_text])
raw = float(np.dot(emb[0], emb[1]) / (np.linalg.norm(emb[0])*np.linalg.norm(emb[1])))
# calibrated 0-1 score
platt = pickle.load(open("platt_calibrator.pkl", "rb"))
score = platt([raw])[0]
Apply smart_truncate_jd (see the research repo, src/text_utils.py) to the job
description first — the model was trained on preprocessed JDs capped at
350 words, and skipping it degrades scores silently.
Limitations
- Match labels are synthetic, generated and hand-curated against a rubric rather than collected from recruiters. Scores reflect that rubric, not hiring outcomes.
- n = 106 on the final test — the confidence intervals above are wide, and differences smaller than their width are not meaningful.
- Not audited for use in automated hiring decisions. See the research repo's data card.
Research repo: https://github.com/dlepighe1/Resume-jd-matcher