sentence_similarity_nepali_v2

A Nepali sentence-embedding model. It maps a Nepali sentence (Devanagari script) to a 768-dimensional vector, so that sentences with similar meaning end up close together under cosine similarity.

It is a Sentence Transformers model fine-tuned from Rajan/NepaliBERT on syubraj/stsb_nepali, a Nepali translation of the STS Benchmark.

Quick start

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("syubraj/sentence_similarity_nepali_v2")

sentences = [
    "रातो, डबल डेकर बस।",
    "रातो डबल डेकर बस।",
    "दुई कालो कुकुर हिउँमा हिंड्दै।",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 768)

# Pairwise cosine similarity, shape [3, 3]
similarities = model.similarity(embeddings, embeddings)
print(similarities)

The model was trained to output a cosine similarity that matches the STS score divided by 5. A score near 1.0 means the two sentences say the same thing, and a score near 0.0 means they are unrelated.

Semantic search

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("syubraj/sentence_similarity_nepali_v2")

query = "एउटा विमान उडिरहेको छ।"
corpus = [
    "हवाई जहाज उडिरहेको छ।",
    "एउटा मान्छे बाँसुरी बजाउँदै छ।",
    "तीन जना चेस खेल्दै छन्।",
    "एउटी महिला मासु फ्राइरहेकी छिन्।",
]

query_embedding = model.encode(query)
corpus_embeddings = model.encode(corpus)

scores = model.similarity(query_embedding, corpus_embeddings)[0]
for score, sentence in sorted(zip(scores.tolist(), corpus), reverse=True):
    print(f"{score:.3f}  {sentence}")

With Transformers only

The pooling step is a mean over token embeddings, weighted by the attention mask.

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "syubraj/sentence_similarity_nepali_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

sentences = ["रातो, डबल डेकर बस।", "रातो डबल डेकर बस।"]
batch = tokenizer(sentences, padding=True, truncation=True, max_length=512, return_tensors="pt")

with torch.no_grad():
    token_embeddings = model(**batch).last_hidden_state

mask = batch["attention_mask"].unsqueeze(-1).float()
embeddings = (token_embeddings * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)
embeddings = F.normalize(embeddings, p=2, dim=1)

print(embeddings @ embeddings.T)  # cosine similarity

Intended uses

• Scoring how similar two Nepali sentences are.
• Semantic search and duplicate or paraphrase detection over short Nepali texts.
• Clustering or grouping Nepali sentences by meaning.
• Sentence features for downstream Nepali classifiers.
• A starting checkpoint for further fine-tuning on your own Nepali pairs.

Out of scope

• Languages other than Nepali, and cross-lingual matching (for example Nepali to English). The model is monolingual.
• Romanized Nepali or heavily code-mixed text. The training data is in Devanagari.
• Long documents. The model was trained on single sentences; see Limitations.
• High-stakes decisions based on a similarity threshold alone, such as plagiarism rulings or legal and medical matching.

Evaluation

Evaluated on the held-out dev set stsb-dev-nepali with Sentence Transformers' EmbeddingSimilarityEvaluator. The figures are for the published weights (the final training checkpoint).

How to read this:

• Use cosine similarity. Dot product is clearly worse because the embeddings are not length-normalised.
• A Spearman correlation of 0.66 is moderate. The model ranks sentence pairs sensibly but not precisely, so treat scores as a ranking signal and tune any threshold on your own data.
• The dev set is a held-out portion of the same translated data, not the official STS-B dev or test split. These numbers are not directly comparable to published STS-B results.

Model details

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_mean_tokens': True})
)

Training

Data

syubraj/stsb_nepali contains 5,749 Nepali sentence pairs translated from stsb_multi_mt, each with a human similarity score from 0 to 5. The sentences come from the STS Benchmark's original domains: image captions, news headlines and forum posts.

• 4,599 pairs (about 80%) were used for training; evaluation uses a held-out dev set.
• Scores were divided by 5 to give labels between 0.0 and 1.0 (mean label 0.54).
• Sentences are short: about 19 tokens on average, 81 at most, in the first 1,000 training samples.

Procedure

Both sentences of a pair are encoded with the same network, and CosineSimilarityLoss (mean squared error between the cosine similarity and the label) is minimised.

Training curve

Most of the gain comes in the first 15 epochs. After that the dev score stays between about 0.65 and 0.67, while the training loss keeps falling from 0.0723 to 0.0007. The model therefore fits the training pairs almost exactly without generalising further, and a much shorter run would give the same quality. The published weights are from the final epoch, not the best one (0.6734 at epoch 41).

Framework versions

• Python 3.10.13
• Sentence Transformers 3.0.0
• Transformers 4.41.2
• PyTorch 2.1.2
• Accelerate 0.30.1
• Datasets 2.19.2
• Tokenizers 0.19.1

Limitations

• Moderate accuracy. A dev-set Spearman of 0.66 leaves room for clear ranking mistakes, especially between pairs of middling similarity.
• Translated training data. The model learned from translated English sentences, some of which read as literal or unnatural Nepali. It has not been evaluated on text written natively in Nepali.
• Narrow domains. Image captions, news headlines and forum posts. Expect weaker results on legal, medical, technical, literary or conversational text.
• Short sentences only. Inputs up to 512 tokens are accepted and longer ones are truncated, but behaviour on paragraphs and documents is untested.
• Symmetric similarity. The model was not trained on question and answer or query and passage pairs, so retrieval with short queries against longer passages may underperform.
• Small dataset, long training. 4,599 pairs for 100 epochs means the model is closely fitted to its training set.
• Inherited bias. Social and cultural biases from the base model's pre-training text and from the STS Benchmark carry over.

Citation

If you use this model, please cite it:

@misc{syubraj2024sentencesimilaritynepaliv2,
  author    = {syubraj},
  title     = {sentence\_similarity\_nepali\_v2},
  year      = {2024},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/3115},
  url       = {https://huggingface.co/syubraj/sentence_similarity_nepali_v2}
}

Sentence-BERT:

@inproceedings{reimers-2019-sentence-bert,
  title     = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
  author    = "Reimers, Nils and Gurevych, Iryna",
  booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
  month     = "11",
  year      = "2019",
  publisher = "Association for Computational Linguistics",
  url       = "https://arxiv.org/abs/1908.10084",
}

STS Benchmark:

@inproceedings{cer-etal-2017-semeval,
  title     = "{S}em{E}val-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation",
  author    = "Cer, Daniel and Diab, Mona and Agirre, Eneko and Lopez-Gazpio, I{\~n}igo and Specia, Lucia",
  booktitle = "Proceedings of the 11th International Workshop on Semantic Evaluation ({S}em{E}val-2017)",
  year      = "2017",
  publisher = "Association for Computational Linguistics",
  url       = "https://aclanthology.org/S17-2001",
}