Model Overview

SecureModernBERT-NER represents a new generation of cybersecurity-focused language models — combining the state-of-the-art architecture of ModernBERT with one of the largest and most diverse CTI-labelled NER corpora ever built.

Unlike conventional NER systems, SecureModernBERT-NER recognises 22 finely-grained, security-specific entity types, covering the full spectrum of cyber-threat intelligence — from THREAT-ACTOR and MALWARE to CVE, IPV4, DOMAIN, and REGISTRY-KEYS.

Trained on more than half a million manually curated spans sourced from real-world threat reports, vulnerability advisories, and incident analyses, it achieves an exceptional balance of accuracy, generalisation, and contextual depth.

This model is designed to parse complex security narratives with human-level precision, extracting both contextual metadata (e.g., ORG, PRODUCT, PLATFORM) and highly technical indicators (e.g., HASHES, URLS, NETWORK ADDRESSES) — all within a single unified framework.

SecureModernBERT-NER sets a new standard for automated CTI entity recognition, enabling the next wave of threat-intelligence automation, enrichment, and analytics.

Quick Start

from transformers import pipeline

model_id = "attack-vector/SecureModernBERT-NER"

pipe = pipeline(
    task="token-classification",
    model=model_id,
    tokenizer=model_id,
    aggregation_strategy="first",
)

text = "TrickBot connects to hxxp://185.222.202.55 to exfiltrate data from Windows hosts."
predictions = pipe(text)
for pred in predictions:
    print(pred)

Sample output:

{'entity_group': 'MALWARE', 'score': np.float32(0.9615546), 'word': 'TrickBot', 'start': 0, 'end': 8}
{'entity_group': 'URL', 'score': np.float32(0.9905957), 'word': ' hxxp://185.222.202.55', 'start': 20, 'end': 42}
{'entity_group': 'PLATFORM', 'score': np.float32(0.92317337), 'word': ' Windows', 'start': 66, 'end': 74}

Intended Use & Limitations

• Use cases: automated tagging of CTI reports, IOC extraction pipelines, knowledge-base enrichment, security-focused RAG systems.
• Languages: English (model was trained and evaluated on English sources only).
• Input format: free-form prose or long-form CTI articles; maximum sequence length 128 tokens during training.
• Limitations: noisy or ambiguous extractions may occur, especially with rare entity types (IPV6, EMAIL) and obfuscated strings. The model does not normalise entities (e.g., deobfuscating hxxp) nor validate indicator authenticity. Always pair with downstream validation and human review.

Training Data

• Size: 502,726 labelled text spans before filtering; 22 distinct entity classes in BIO format.
• Label distribution (spans): ORG (approx. 198k), PRODUCT (approx. 79k), MALWARE (approx. 67k), PLATFORM (approx. 57k), THREAT-ACTOR (approx. 49k), SERVICE (approx. 46k), CVE (approx. 41k), LOC (approx. 38k), SECTOR (approx. 34k), TOOL (approx. 29k), plus indicator types such as URL, IPV4, SHA256, MD5, and REGISTRY-KEYS.
• Pre-processing: JSONL articles were tokenised and converted to BIO tags; spans in conflict were resolved manually and via automated heuristics before upload.

Label Mapping

Training Procedure

• Base model: answerdotai/ModernBERT-large.
• Hardware: single Nvidia L40S instance (8 vCPU / 62 GB RAM / 48 GB VRAM).
• Optimisation setup: mixed precision fp16, optimiser adamw_torch, cosine learning-rate scheduler, gradient accumulation 1.
• Key hyperparameters: learning rate 5e-5, batch size 128, epochs 5, maximum sequence length 128.

Evaluation

AutoTrain reports the following micro-averaged metrics on its validation split (seqeval entity scoring):

An independent re-evaluation against a consolidated CTI set (same taxonomy as this model) produced the label-level accuracy breakdown below. These scores are macro-averaged across labels and therefore are not numerically comparable to the micro metrics above, but they provide insight into class balance and span quality.

• Macro accuracy: 0.8776

Because micro vs macro averaging and dataset composition differ, expect numerical gaps between the two evaluations even though both describe the same checkpoint.

These metrics were computed with the seqeval micro-average at the entity level.

External Benchmarks

The following tables report detailed results on a shared CTI validation set. Do not compare the per-label values across models directly: each checkpoint uses a different taxonomy or remapping strategy, so accuracy percentages can be misleading when labels are aligned or collapsed differently. Use the per-model tables to understand performance within a single schema, and interpret macro-accuracy scores with caution.

CyberPeace-Institute/SecureBERT-NER

• Macro accuracy: 0.3820

PranavaKailash/CyNER-2.0-DeBERTa-v3-base

• Macro accuracy: 0.4348

cisco-ai/SecureBERT2.0-NER

• Macro accuracy: 0.6100

Responsible Use

• Confirm entity detections before acting on indicators (e.g., automated blocking).
• Combine with enrichment and scoring systems to filter false positives.
• Monitor for drift if applying to new domains (e.g., non-English sources, informal channels).
• Respect licensing and confidentiality of any proprietary CTI sources used for inference.

Support & Connect

• ❤️ Like the repo if you found it useful
• ☕ Support me: Say thanks by buying me a coffee! https://buymeacoffee.com/juanmcristobal
• 💼 Open to work: https://www.linkedin.com/in/jmcristobal/

If you use SecureModernBERT-NER in a project, feel free to share it in the Discussions/Issues — I love seeing real-world use cases.

Citation

If you find this model useful, please cite the repository and the base model:

@software{securemodernbert_ner_2025,
  author = {Juan Manuel Cristóbal Moreno},
  title = {SecureModernBERT-NER: Cyber Threat Intelligence Named Entity Recogniser},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/attack-vector/SecureModernBERT-NER}
}

Contact

Questions or feedback? Open an issue on the Hugging Face model repository or reach out at @juanmcristobal.