cti-ner-roberta-large (clean LLM-gold retrain, v2)
roberta-large fine-tuned as a BIO token classifier for cyber threat
intelligence NER on CISA advisories: 16 entity types (attack technique
id/name, CVE/CWE identifiers, file hash/name/path, IP address, email
address, malware family, threat actor group, hacking tool, tactic, TLP
classification, YARA rule name, affected software product).
This version replaces the earlier checkpoint trained on dictionary-silver labels. Training gold is LLM-proposed (Claude Sonnet 5, exhaustive extraction under per-label span-format validators) with the evaluation split human-curated. See the paper in the repository below for the methodology and the circularity analysis that motivated the switch.
Evaluation (measured on this checkpoint)
Document-level span scoring on the held-out 8-advisory test split
(strict = exact span + label), verified independently with the repository
harness (ner.finetune_roberta --eval-only):
| Metric | Strict | Relaxed |
|---|---|---|
| Micro F1 | 0.63 | 0.69 |
| Macro F1 | 0.71 | 0.75 |
The paper's committed result for this architecture
(data/gold_v2/results/trained-roberta-large.json) is 0.648 strict micro
from an earlier training run; this checkpoint is a re-run with identical
data and hyperparameters, and the 1.4-point difference is seed/hardware
variance on an 8-document test set. Either way the architecture ordering
holds: SecureBERT 0.71 $>$ RoBERTa-base 0.659 $>$ RoBERTa-large $>$
GLiNER 0.560 on the identical gold and harness, i.e., domain-adaptive
pretraining beats raw parameter count on this task.
Training details
- Data: 66 training advisories (2,799 chunks), 8 val, 8 test; document-level
split committed in the repository (
data/split_slugs.json). - 8 epochs, lr 2e-5, seed 0, wordpiece-budget sliding windows; best epoch
by validation strict F1. Trained on Colab CUDA via
notebooks/RoBERTa_Large_CTI_Colab.ipynb(corpus embedded in the notebook).
Usage
Inference should replicate the sliding-window word-tagging procedure in
ner/finetune_roberta.py (see notebooks/hf_models_demo.ipynb in the
repository for a worked example); a naive single-pass tokenizer call
truncates long, hash-heavy advisory text.
Links
- Code, gold data, paper: https://github.com/dandye/nlp_capstone
- Companion models:
pzryathzsdhc/cti-ner-securebert,pzryathzsdhc/cti-ner-gliner