vulrank-span-model
A token classifier (fine-tuned microsoft/deberta-v3-base) that tags software
vulnerability advisory text for four claim types: exploitation, patch, poc,
affected_function_api. It is the local, no-API-key proposer for
VulRank — a grounded vulnerability-triage tool.
How the training labels were produced
Labels were not hand-annotated. They were generated by having claude-sonnet-5
propose candidate claim spans over ~10,000 PyPI OSV advisories, then running every
proposal through VulRank's own deterministic verbatim gate
(vulrank.extract.discovery.gate.ground_candidate): a proposed span survives only if
it is an exact substring of the (normalized) advisory text. Anything paraphrased,
summarized, or hallucinated by the teacher model was discarded before this model ever
saw it.
- Teacher:
claude-sonnet-5(Anthropic), via the Batch API. - Corpus: 10,000 PyPI OSV advisories -> 9,334 successfully labeled -> 8,020 advisories with at least one gate-verified span (17,376 spans) after gating and cleanup.
- Student:
microsoft/deberta-v3-base, 9-label BIO token classification (O+B-/I-x the four claim types), 4 epochs, 90/10 split,max_length=512.
Scope of the verbatim guarantee: the span text is verbatim-verified by construction. The type label is not — for ~85% of spans the classic lexicon does not re-fire, so the type falls back to the teacher's own unverified judgment. This is by design (see "what a prediction means" below): types are proposals, and VulRank re-derives them at runtime rather than trusting either the teacher's or this model's.
Label cleanup
Four issues were found by auditing the gold spans, and fixed before this run:
| fix | effect |
|---|---|
| Trim markdown boundary noise (backticks, quotes, emphasis) from span edges | 774 spans trimmed; eliminates predictions like a lone ` or ) |
| Resolve overlapping spans (keep shortest/most specific) | 675 dropped; BIO is flat and cannot encode two types over one token |
Drop patch spans that state an affected version range, not a fix | 1,036 dropped — 24% of the patch class was mistyped |
Raise max_length 256 -> 512 | label loss to truncation: 10.7% -> 2.2% |
Evaluation (held-out split)
| Macro F1 | Span-level recall | |
|---|---|---|
| Classic lexicon alone (no model) | — | 0.9% |
| This model | 0.723 | 75.2% |
Per-class F1 versus the previous release (trained on uncleaned labels at
max_length=256), each verified stable across two random seeds:
| label | previous | this model |
|---|---|---|
B-patch | 0.805 | 0.874 |
B-affected_function_api | 0.456 | 0.531 |
B-exploitation | 0.719 | 0.723 |
poc is the smallest class (~47 span-starts in the validation split) and its F1 swings
±0.12 on seed change alone; treat any single poc number as noise.
Numbers are not comparable across releases in absolute terms — the label cleanup changed the gold set, so each release is scored on a different validation split (the classic-lexicon baseline on each split, reported above, is a rough difficulty proxy). Per-class deltas verified across seeds are the meaningful signal.
Important: what a prediction from this model means
A span this model proposes is not verified truth. In VulRank, every prediction is
re-run through the same deterministic gate used to build its training data: the
verbatim-substring check, rule-based re-polarization (negation.polarity_of), and
rule-based re-typing (the lexicon). This model's own type and confidence are never
trusted directly — it only proposes candidate spans; rules dispose. If you consume this
model outside VulRank, treat its output as a high-recall candidate generator, not a
verified claim extractor.
Usage
Not intended for standalone use — designed to be loaded by
vulrank.extract.discovery.local_client.LocalSpanClient via
pip install 'vulrank[local]'. See the
VulRank repo for details.