jobbert-ner-sonnet-v2

Named entity recognition (NER) model for English-language job postings, distilled from LLM labels: one of the ten students released with the paper A Scalable LLM-Distilled Named Entity Recognition Framework for Labour-Market Analytics Using Apache Spark (Soltani and Hanine, 2026).

• Student: S3′ (s3_jobbert_chunked_sonnet)
• Teacher: Claude Sonnet 4.6, labels obtained through AWS Bedrock
• Architecture: jjzha/jobbert-base-cased fine-tuned on 450-token windows with a 225-token stride
• Download size: about 432 MB

The v2 version of S3, retrained on overlapping 450-token windows so that the whole posting is in distribution. It gains +0.044 F1 over sliding-window inference on the v1 weights (+0.028 over single-shot S3) but closes only about 17–20 % of the gap to spaCy; the residual concentrates on the soft-boundary types (SKILL, CERT, EDUCATION, EXPERIENCE_LEVEL), which points to the token-level training objective rather than to input coverage.

Intended use

Entity extraction from English-language job-posting descriptions into an eight-type schema:

SKILL, JOB_TITLE, COMPANY, LOCATION, EXPERIENCE_LEVEL, EDUCATION, CERT, COMPENSATION.

Appropriate uses include indexing postings for search and analytics, aggregating skill demand for labour-market research, benchmarking the cost, quality and speed of distilled NER, and teaching NLP or distillation.

Out-of-scope use

Not suitable for:

• CVs or résumés (a different register; use a model trained on CVs).
• Non-English postings, or postings from locales the teacher labels did not represent.
• Fully automated candidate screening or hiring decisions; any downstream ranking or filtering needs an application-side schema and bias review first (see Ethical considerations).
• Medical, legal, financial or other high-stakes decision support.

Training

• Teacher labels: 5,000 stratified postings labelled by Claude Sonnet 4.6 at temperature 0; both teachers labelled the same 5,000 postings. During this run max_tokens was raised from 4,096 to 8,192 after two entity-dense postings were truncated, and both were re-run.
• Split: 80/10/10 train/dev/test by md5(job_link) mod 10, identical for both teachers. Students train on the train partition and the gold postings come from the dev partition, so no student is evaluated on a posting it was trained on.
• Model: jjzha/jobbert-base-cased fine-tuned for token classification over 17 BIO labels on 450-token windows with a 225-token stride, one training example per window (an entity that straddles a window boundary is dropped from that window); AdamW, learning rate 2e-5 with 10 % linear warm-up, weight decay 0.01, batch size 16, 3 epochs with the best epoch kept by dev F1, fp16, seed 42.
• Hardware: one NVIDIA A10G 24 GB GPU (AWS g5.xlarge).
• Specification and training script: pipeline/training/experiments/specs/s3_jobbert_chunked_sonnet.yaml and pipeline/scripts/train_jobbert_chunked.py in the project repository.

Evaluation

The gold set holds 516 postings from the dev partition, annotated by a human against written guidelines. This student is scored on all 516 postings. The metric is micro-F1 over per-posting sets of (text, type) pairs: positions are ignored and a repeated mention counts once. Intervals come from 10,000 posting-level bootstrap resamples (seed 42).

Latency figures are indicative only. They were not measured on a common execution provider (the dense spaCy and JobBERT figures come from GPU-served evaluation, the ONNX int8 figures from CPU), so they rank neither architectures nor quantisation. The deployment-relevant figure is S2's mean of 86.4 ms per posting inside the executors of the paper's 1.3M-posting Spark run on CPU instances.

Per-entity-type scores

Same windowed evaluation as the figure above.

Teacher comparison

The teacher, Claude Sonnet 4.6, scores 0.517 [0.503, 0.530] on the same gold postings.

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("AchrafSoltani/jobbert-ner-sonnet-v2")
model = AutoModelForTokenClassification.from_pretrained("AchrafSoltani/jobbert-ner-sonnet-v2")
ner = pipeline("token-classification", model=model, tokenizer=tokenizer,
               aggregation_strategy="simple")

text = 'Senior Machine Learning Engineer at Acme Corp in Berlin. Requires 5+ years of experience with PyTorch, AWS, and Kubernetes. MSc in Computer Science preferred. Salary $140,000 – $180,000.'
for ent in ner(text):
    print(ent["word"], "->", ent["entity_group"], round(float(ent["score"]), 2))

# Output on this release (checked on 2026-10-10 with transformers 4.57.6):
# Senior Machine Learning Engineer     -> JOB_TITLE 0.79
# Acme Corp                            -> COMPANY 0.94
# Berlin                               -> LOCATION 0.84
# 5 + years of                         -> EXPERIENCE_LEVEL 0.87
# PyTorch                              -> SKILL 0.96
# AWS                                  -> SKILL 0.94
# Kubernetes                           -> SKILL 0.93
# MSc in Computer Science              -> EDUCATION 0.96
# $ 140, 000 – $ 180, 000              -> COMPENSATION 0.97
#
# WordPiece decoding inserts spaces into numeric spans ('$ 140, 000').
# A single call reads at most 512 tokens. For full postings, apply the windowed inference
# of pipeline/scripts/bootstrap_ci_chunked.py in the project repository, which the
# reported scores use (450-token windows, 225-token stride).

Ethical considerations

This model extracts entities from job postings, whose downstream consumers are typically hiring, ranking or matching systems. Three cautions from the paper apply:

• Schema-induced bias. SKILL over-extraction is inherited from the LLM teacher: soft-skill phrases ("communication skills", "interpersonal skills") and generic tools ("Excel", "CRM") are over-represented relative to the gold standard. A downstream ranker that treats such phrases as filters encodes the teacher's lexical habits as a hiring criterion and should not be built without a schema review at the application layer.
• Contested ground truth. The paper compared S2's extracted skills with the skills field distributed with the Kaggle release (job_skills.csv; its provenance is undocumented) on the 938,028 postings that have both a Kaggle skills list and at least one S2 SKILL span. Only 9.6 % of that field's skills are also extracted by S2, and 56.4 % of S2's skills are absent from the field. Neither is a ground truth; the numbers measure schema divergence, not model quality.
• Consent and licensing. The training corpus is a publicly released Kaggle redistribution of scraped LinkedIn postings. People named in postings, such as recruiters and hiring managers, did not consent to their postings being reprocessed for research. The model is licensed CC BY-NC 4.0 for research and non-commercial evaluation only; any commercial deployment needs a separate legal and ethical review of the data-provenance chain.

Limitations

• Trained and evaluated on English-language LinkedIn postings from a publicly released 2024 Kaggle redistribution; generalisation to other platforms or languages is unevaluated.
• The gold set has a single human annotator and no second human annotation, so read the F1 against the label ambiguity of soft-boundary types such as SKILL.
• The output schema is fixed to the eight types above; finer-grained or taxonomy-aligned schemas need retraining on new labels.
• Full-text coverage comes from windowed inference, so compute grows linearly with posting length; for latency-sensitive full-text extraction, prefer the spaCy students.

Citation

@unpublished{soltani2026distilledner,
  author = {Achraf Soltani and Mohamed Hanine},
  title  = {A Scalable {LLM}-Distilled Named Entity Recognition Framework for Labour-Market Analytics Using {Apache Spark}},
  year   = {2026},
  note   = {Manuscript},
  url    = {https://github.com/achrafsoltani/distributed-ner-on-spark},
}

The JobBERT base model:

@inproceedings{zhang-etal-2022-skillspan,
  title     = {{S}kill{S}pan: Hard and Soft Skill Extraction from {E}nglish Job Postings},
  author    = {Zhang, Mike and Jensen, Kristian N{\o}rgaard and Sonniks, Sif and Plank, Barbara},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year      = {2022},
  address   = {Seattle, United States},
  publisher = {Association for Computational Linguistics},
  pages     = {4962--4984},
  doi       = {10.18653/v1/2022.naacl-main.366},
}

Licence

• Model weights: CC BY-NC 4.0, for research and non-commercial evaluation only.
• Code in the project repository: Apache 2.0; the gold annotations and documentation: CC BY-NC 4.0.
• Base model: jjzha/jobbert-base-cased (Zhang et al., 2022). Its Hub card states no licence; check its terms before reuse beyond research.