SemCSE Model Card

The SemCSE model is an embedding model for scientific abstracts and sentences in general, usable for clustering, retrieval, and many other embedding-related applications. The novelty of our approach is the focus on embeddings that accurately reflect the semantics of the paper, which was lacking for many existing approaches that were trained using citation information.

The novel, semantically-oriented training procedure leads to state-of-the-art resutls on our novel semantic embedding benchmark (please see our paper for details), as well as to state-of-the-art results for models of its size on the established SciRepEval benchmark.

Note that this model uses cosine similarity for computing similarity in its embedding space. Our main model instead uses Euclidean distance.

Model Details

Model Description

• Developed by: CLAUSE group at Bielefeld University
• Model type: DeBERTa v2
• Languages: Mostly english
• Finetuned from model: KISTI-AI/Scideberta-full

Model Sources

• Repository: github.com/inas-argumentation/SemCSE
• Paper: https://arxiv.org/abs/2507.13105

How to Get Started with the Model

Minimal example on how to create embeddings with our model:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("CLAUSE-Bielefeld/SemCSE_cosine")
model = AutoModel.from_pretrained("CLAUSE-Bielefeld/SemCSE_cosine")

text = "Your text to be embedded."
batch = tokenizer([text], return_tensors="pt")
embedding = model(**batch)["last_hidden_state"][0, 0]

Training Details

This model was trained on a dataset of summaries for 350K scientific abstracts from various domains. We used a triplet loss to encourage summaries of the same abstract to be placed nearby in the embedding space. The dataset and exact training procedure can be found in our GitHub repo,

Evaluation

We introduce a novel semantic scientific embedding benchmark:

Notes:

• Bold = Best result
• Underlined = Second-best result
• ↓ = Lower is better (ranking-based tasks)
• ↑ = Higher is better (clustering and overall performance)

We also evaluate SemCSE on the SciRepEval benchmark:

Notes:

• Bold = Best result
• Underlined = Second-best result
• ↑ = Higher is better

Citation

BibTeX:

@misc{brinner2025semcsesemanticcontrastivesentence,
      title={SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts}, 
      author={Marc Brinner and Sina Zarriess},
      year={2025},
      eprint={2507.13105},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.13105}, 
}