SymSTS-MiniLM

1. What This Model Is

SymSTS-MiniLM-L6 is a fine-tuned version of sentence-transformers/all-MiniLM-L6-v2, specialized for Semantic Textual Similarity (STS).

The key idea behind this model is symmetric data augmentation. During training, every sentence pair (A, B, score) is duplicated as (B, A, score). This forces the model to learn that similarity is bidirectional: similarity(A, B) = similarity(B, A). This simple technique produces consistent gains across all historical STS benchmarks.

The model has 22.7 million parameters, runs fast on CPU, and fits comfortably on consumer GPUs.

This is NOT a general-purpose embedding model. It is built for one job: measuring how similar two sentences are in meaning. It is not optimized for document retrieval, classification, or multilingual tasks.

2. Model Architecture

3. Intended Use

4. Benchmark Results

All scores are Spearman rank correlation measured locally using the MTEB library on official test splits.

4.1 STS Results vs Base Model

4.2 Honest Comparison Against Other Models

Scores for external models are approximate, taken from the MTEB leaderboard and published model cards. Our scores are measured locally. We include models that outperform ours for full transparency.

Summary: SymSTS-MiniLM-L6 outperforms all models at or below 33M parameters and several models up to 109M parameters on STS tasks. It is outperformed by larger base-architecture models (110M+) and LLM-based embeddings, which is expected given the parameter gap.

5. Training Details

5.1 Loss Function

CosineSimilarityLoss — minimizes the mean squared error between the predicted cosine similarity and the human-annotated similarity score.

5.2 Symmetric Data Augmentation

For every training pair: Original: (Sentence_A, Sentence_B, score) Augmented: (Sentence_B, Sentence_A, score)

Both are included, doubling the effective training set from ~8k to ~16k pairs. Note: because cosine similarity is symmetric by construction (sim(A,B) = sim(B,A) regardless of training), this augmentation's main effect is increasing the volume of training pairs rather than teaching the model a new bidirectional property. The performance gains reported in Section 4 are real and measured directly; the ablation isolating "more data" from "swap specifically" is planned as future work.

5.3 Hyperparameters

6. Training Data

The model was fine-tuned on the train splits of the following datasets. STS13 through STS16 and SICK-R were not used during training. They are held-out evaluation benchmarks only.

The raw dataset files are not redistributed in this repository. Only the fine-tuned model weights are provided.

7. Usage

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("blueprint-ai/SymSTS-MiniLM")

sentences = [
    "The cat sits on the mat.",
    "A feline is resting on the rug.",
    "The stock market crashed today."
]

embeddings = model.encode(sentences, normalize_embeddings=True)

sim_01 = util.cos_sim(embeddings[0], embeddings[1]).item()
sim_02 = util.cos_sim(embeddings[0], embeddings[2]).item()

print(f"Cat vs Feline:  {sim_01:.4f}")
print(f"Cat vs Stocks:  {sim_02:.4f}")
#8. Full Training Code
This is the exact script used to train SymSTS-MiniLM-L6.
import warnings
warnings.filterwarnings("ignore")

import os
import torch
from datasets import load_dataset, Dataset

from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    InputExample,
)
from sentence_transformers.losses import CosineSimilarityLoss
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator


# =============================================================
# Configuration
# =============================================================
BASE_MODEL    = "sentence-transformers/all-MiniLM-L6-v2"
OUTPUT_DIR    = "./output/symsts-minilm-l6"
RUNS_DIR      = "./output/runs-symsts-minilm-l6"
MAX_SEQ_LEN   = 256
BATCH_SIZE    = 16
LEARNING_RATE = 1e-5
EPOCHS        = 2

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")


# =============================================================
# Helpers
# =============================================================
def clean_text(text):
    if text is None:
        return ""
    return " ".join(str(text).strip().split())


def get_main_score(result):
    if isinstance(result, dict):
        for key, value in result.items():
            if "spearman_cosine" in key:
                try: return float(value)
                except Exception: pass
        for key, value in result.items():
            if "spearman" in key:
                try: return float(value)
                except Exception: pass
        for value in result.values():
            try: return float(value)
            except Exception: pass
    try: return float(result)
    except Exception: return 0.0


# =============================================================
# 1. Load Base Model
# =============================================================
print("Loading base model...")
student = SentenceTransformer(BASE_MODEL, device=device)
student.max_seq_length = MAX_SEQ_LEN


# =============================================================
# 2. Load STS Training Data with Symmetric Augmentation
# =============================================================
print("Loading STS training data...")

sources = [
    ("mteb/stsbenchmark-sts", "train"),
    ("mteb/sts12-sts",        "train"),
]

sentence1_list = []
sentence2_list = []
label_list     = []
seen_pairs     = set()


def add_pair(s1, s2, score):
    s1 = clean_text(s1)
    s2 = clean_text(s2)
    if not s1 or not s2:
        return

    key = tuple(sorted((s1.lower(), s2.lower())))
    if key in seen_pairs:
        return
    seen_pairs.add(key)

    # Forward
    sentence1_list.append(s1)
    sentence2_list.append(s2)
    label_list.append(score)

    # Reverse (Symmetric Augmentation)
    if s1.lower() != s2.lower():
        sentence1_list.append(s2)
        sentence2_list.append(s1)
        label_list.append(score)


for dataset_name, split in sources:
    try:
        ds = load_dataset(dataset_name, split=split)
        count = 0
        for row in ds:
            s1    = row.get("sentence1")
            s2    = row.get("sentence2")
            score = row.get("score", 0.0)
            try:
                score = float(score)
            except Exception:
                continue
            if score > 1.0:
                score = score / 5.0
            score = max(0.0, min(1.0, score))
            add_pair(s1, s2, score)
            count += 1
        print(f"  Loaded {count} original pairs from {dataset_name}")
    except Exception as e:
        print(f"  Skipping {dataset_name}: {e}")

print(f"Unique pairs: {len(seen_pairs)}")
print(f"Total augmented rows: {len(sentence1_list)}")

train_dataset = Dataset.from_dict({
    "sentence1": sentence1_list,
    "sentence2": sentence2_list,
    "label":     label_list,
})


# =============================================================
# 3. STS-B Test Evaluator
# =============================================================
print("Loading STS-B test evaluator...")

sts_test = load_dataset("mteb/stsbenchmark-sts", split="test")
eval_examples = []
for row in sts_test:
    s1    = clean_text(row.get("sentence1"))
    s2    = clean_text(row.get("sentence2"))
    score = float(row.get("score", 0.0)) / 5.0
    eval_examples.append(InputExample(texts=[s1, s2], label=score))

evaluator = EmbeddingSimilarityEvaluator.from_input_examples(
    eval_examples,
    name="sts-b-test",
)

base_score = get_main_score(evaluator(student))
print(f"Base model STS-B score: {base_score:.4f}")


# =============================================================
# 4. Train
# =============================================================
print("Training SymSTS-MiniLM-L6...")

train_loss = CosineSimilarityLoss(model=student)

training_args = SentenceTransformerTrainingArguments(
    output_dir                  = RUNS_DIR,
    num_train_epochs            = EPOCHS,
    per_device_train_batch_size = BATCH_SIZE,
    per_device_eval_batch_size  = BATCH_SIZE,
    learning_rate               = LEARNING_RATE,
    warmup_steps                = 0.1,
    weight_decay                = 0.01,
    fp16                        = torch.cuda.is_available(),
    bf16                        = False,
    logging_steps               = 50,
    save_strategy               = "no",
    eval_strategy               = "epoch",
    dataloader_num_workers      = 0,
    report_to                   = "none",
    remove_unused_columns       = False,
    seed                        = 42,
)

trainer = SentenceTransformerTrainer(
    model         = student,
    args          = training_args,
    train_dataset = train_dataset,
    loss          = train_loss,
    evaluator     = evaluator,
)

trainer.train()

os.makedirs(OUTPUT_DIR, exist_ok=True)
try:
    trainer.save_model(OUTPUT_DIR)
except Exception:
    student.save_pretrained(OUTPUT_DIR)

print(f"Model saved to: {OUTPUT_DIR}")


# =============================================================
# 5. Final Evaluation
# =============================================================
final_score = get_main_score(evaluator(student))

print("=" * 50)
print(f"Base model STS-B:   {base_score:.4f}")
print(f"SymSTS STS-B:       {final_score:.4f}")
print(f"Improvement:        {final_score - base_score:+.4f}")
print("=" * 50)

#9. Evaluation Code
This is the script used to produce the benchmark tables above.
import warnings
warnings.filterwarnings("ignore")

import os, json, glob, mteb
from sentence_transformers import SentenceTransformer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

TASKS = [
    "STS12", "STS13", "STS14", "STS15",
    "STS16", "STSBenchmark", "SICK-R",
]

def get_scores(model_path, out_dir):
    model      = SentenceTransformer(model_path)
    tasks      = mteb.get_tasks(tasks=TASKS)
    evaluation = mteb.MTEB(tasks=tasks)
    evaluation.run(model, output_folder=out_dir, verbosity=0)

    scores = {}
    for task_name in TASKS:
        files = glob.glob(f"{out_dir}/**/*{task_name}*.json", recursive=True)
        if files:
            with open(files[0]) as f:
                d = json.load(f)
                if "scores" in d and "test" in d["scores"]:
                    scores[task_name] = d["scores"]["test"][0].get("main_score", 0.0)
    return scores

base = get_scores("sentence-transformers/all-MiniLM-L6-v2", "./mteb_base")
ours = get_scores("./output/symsts-minilm-l6",              "./mteb_ours")

print(f"{'Task':<16} {'Base':>8} {'Ours':>8} {'Diff':>8}")
print("-" * 44)
for t in TASKS:
    b = base.get(t, 0)
    o = ours.get(t, 0)
    print(f"{t:<16} {b:>8.4f} {o:>8.4f} {o-b:>+8.4f}")

10. Limitations

Trained on a small curated dataset (~16,000 augmented pairs). May not generalize well to domains far from the STS benchmark distribution (news, forums, headlines, image captions). Symmetric augmentation assumes similarity is perfectly symmetric. This is generally true for semantic similarity but may not hold for all retrieval scenarios. Inherits biases from the base MiniLM architecture and SemEval training data, which is predominantly English, web-sourced text. Should not be used as the sole decision-making system in high-stakes applications without human oversight.

11. License

This fine-tuned model is released under the Apache 2.0 License, consistent with the license of the base model sentence-transformers/all-MiniLM-L6-v2. The training datasets (STS Benchmark, STS12) are released for research and evaluation purposes by their respective authors. The raw dataset files are not redistributed here.

12. Credits and Citations

Base Model: Nils Reimers and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of EMNLP 2019. Model Card STS Benchmark: Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia. "SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation." Proceedings of SemEval-2017. STS12: Eneko Agirre, Daniel Cer, Mona Diab, Inigo Lopez-Gazpio, Lucia Specia. "SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity." Proceedings of *SEM 2012. Evaluation Framework: MTEB: Massive Text Embedding Benchmark

13. Contact

For questions, bug reports, or collaboration inquiries: Email: blueprintai.help1@gmail.com