Semantic Highlighting Classifier (ModernBERT + CRF)

Tags rhetorical-role spans in technical/academic abstracts -- Contribution, Method, Result, Evidence, Limitation, FutureWork, Safety -- for a semantic highlighting accessibility tool aimed at readers with dyslexia/ADHD. Instead of highlighting keywords, it labels what role each span of text plays, so a reader can skim a paper by color rather than reading every word.

Architecture: a ModernBERT-base encoder with a linear classification head, decoded with a CRF (Conditional Random Field) layer instead of independent per-token argmax. The CRF was added specifically after an earlier version of this model was found to change its predicted label on nearly every token (even splitting labels mid-word) -- independent per-token classification has no mechanism to penalize that, while a CRF explicitly learns which label transitions are plausible and decodes the whole sequence jointly. Full diagnosis and before/after examples in the project's findings log.

This is the model used by the interactive Gradio demo in the project repo (app.py, run locally with uv run python app.py or make demo) and is the fast, real-time-capable option in this project (see the paired Qwen2.5-1.5B LoRA adapter for a generative alternative that scores higher but is ~20x+ slower and only reliable on a minority of inputs -- both models and the full comparison are documented in the findings log linked above).

Usage

This is a custom architecture (not a standard transformers AutoModelForTokenClassification, since it wraps a CRF decoding step), so it needs the model class from the project repo rather than transformers alone:

import torch
from transformers import AutoTokenizer
# pip install the project, or copy classifier/crf_model.py + classifier/labels.py
from semantic_highlighting.classifier.crf_model import ModernBertCrfForTokenClassification
from semantic_highlighting.classifier.predict import predict_spans_crf

tokenizer = AutoTokenizer.from_pretrained("Rychanfox/semantic-highlighting-modernbert-crf")
model = ModernBertCrfForTokenClassification.from_pretrained("Rychanfox/semantic-highlighting-modernbert-crf")
model.eval()

text = "We propose a new method for X. It achieves strong results using only 12 minutes of data."
spans = predict_spans_crf(text, model, tokenizer)
for span in spans:
    print(span.label, repr(span.text))

For real documents (not just single short abstracts), see deploy/chunking.py in the repo for sentence-aware chunking plus a postprocessing pipeline (merging fragments, fixing boundaries, capping total highlighted coverage to ~20% of the document) that the raw model output on its own does not do.

Training data

Trained on Rychanfox/semantic-highlighting-abstracts -- arXiv abstracts weak-labeled by a local LLM and partially hand-corrected. See that dataset's card for the full collection/cleaning methodology.

Evaluation

Exact-match span-level F1 against a 21-abstract hand-corrected gold set (small -- treat these numbers as directional, not precise):

For context: a zero-training-cost baseline (frozen sentence embeddings + logistic regression) scores 0.14/0.10 on the same metrics -- the fine-tuning, CRF architecture, and data-cleaning work in this project bought a real but modest improvement over a baseline that trains in about a second. That's an intentionally reported honest comparison, not a caveat to bury -- full context in the findings log.

Limitations

• Small gold evaluation set (21 abstracts) -- reported numbers carry real variance.
• Exact-match performance is modest in absolute terms; character-overlap (does it highlight roughly the right region) is much stronger (0.86) than exact-match (does it get the precise boundary right).
• Trained predominantly on ML/NLP arXiv abstracts; domain transfer to other fields or full paper bodies (vs. abstracts) is untested.
• Safety and FutureWork labels are rare in the training data and the model underperforms on them specifically.