SentenceTransformer based on mjaliz/bslm-pair-206k

This is a sentence-transformers model finetuned from mjaliz/bslm-pair-206k on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mjaliz/bslm-pair-206k
• Maximum Sequence Length: 96 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'لباس هودی',
]
documents = [
    'هودی مشکی طرح رونالدینیو لباس بارسلونا FC Barcelona',
    'هودی رونالدینیو با لباس بارسلونا و ژست خاص - کد 11311',
    'p-33828382',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7550, 0.7623, 0.2425]])

Evaluation

Metrics

Information Retrieval

• Dataset: dev
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

json

• Dataset: json
• Size: 2,991,559 training samples
• Columns: anchor, rewritten_title, original_title, original_usable, product_group_id, product_id, and query_id
• Approximate statistics based on the first 100 samples:

  	anchor	rewritten_title	original_title	original_usable	product_group_id	product_id	query_id
  type	string	string	string	bool	int	string	string
  modality	text	text	text			text	text
  details	min: 5 tokensmean: 7.99 tokensmax: 15 tokens	min: 12 tokensmean: 22.05 tokensmax: 41 tokens	min: 5 tokensmean: 14.7 tokensmax: 33 tokens	False: ~5.77%True: ~94.23%	1030862751937536: ~0.96%119064850511069427: ~0.96%192511709702592144: ~0.96%263328678771954932: ~0.96%291520852322212125: ~0.96%472385400846710711: ~0.96%484401372862644547: ~0.96%618531566146852538: ~0.96%647831332662481881: ~0.96%780967915604863284: ~0.96%903012556598518432: ~0.96%903425124187090858: ~0.96%996904868366954223: ~0.96%1146465365284622564: ~0.96%1210638753800252501: ~0.96%1367831584896500957: ~0.96%1499769885652000917: ~0.96%1829307944545599193: ~0.96%1859407507160763064: ~0.96%1877799178259742051: ~0.96%2006189160986889749: ~0.96%2010942333309107070: ~0.96%2038817785620382804: ~0.96%2103787562834935779: ~0.96%2112745891879077907: ~0.96%2218763222356756464: ~0.96%2442015172761774287: ~0.96%2444932559308539681: ~0.96%2537845397179826724: ~0.96%2594279159299525477: ~0.96%2730965935514536189: ~0.96%2786159111090846965: ~0.96%2867076061669685477: ~0.96%2887751882743475269: ~0.96%2999913798983822885: ~0.96%3005267473463394716: ~0.96%3046617812543140456: ~0.96%3125258495322919942: ~0.96%3224712290142852358: ~0.96%3425140391563133879: ~0.96%3428136546146965774: ~0.96%3545788960488691120: ~0.96%3614501397124292149: ~0.96%3682142006856061095: ~0.96%3811572781835714604: ~0.96%3812360642580912795: ~0.96%3822785927866402625: ~0.96%3829023088209848122: ~0.96%3844191058368484464: ~0.96%4084615514929200962: ~0.96%4096474777489716933: ~0.96%4114927997369175255: ~0.96%4231982427553068089: ~0.96%4279369369456856192: ~0.96%4378380396302367059: ~0.96%4384231002193651947: ~0.96%4600858132389458664: ~0.96%4695111211373069683: ~0.96%4878945016725597997: ~0.96%4910605022759678037: ~0.96%4928056820571584525: ~0.96%5033948001284848237: ~0.96%5416455593172543525: ~0.96%5798429995041690589: ~0.96%5829597460078251848: ~0.96%5860553625479554861: ~0.96%5870234672202558664: ~0.96%5921085837518525024: ~0.96%5934580814343084429: ~0.96%5947338821078599400: ~0.96%5992059194996137848: ~0.96%6291923032141321752: ~0.96%6534426692390445382: ~0.96%6568522923892579353: ~0.96%6692829788266165061: ~0.96%6761563738039513383: ~0.96%6844668126458996158: ~0.96%6877987885922504233: ~0.96%7230470559738412225: ~0.96%7233430298467283068: ~0.96%7301544072977166341: ~0.96%7342483594400532132: ~0.96%7355873796043378090: ~0.96%7393488442012240316: ~0.96%7597657601368596767: ~0.96%7642761448828281767: ~0.96%7878086468019751783: ~0.96%7903444671292700660: ~0.96%7964437025590389389: ~0.96%8039660886712821289: ~0.96%8071508475710436468: ~0.96%8280312650043386963: ~0.96%8324485470027659419: ~0.96%8359689853251234944: ~0.96%8471609307232547258: ~0.96%8503144545886180400: ~0.96%8728905541838842737: ~0.96%8821089739081881067: ~0.96%8830839714534070656: ~0.96%9016168636501230322: ~0.96%9037275121370260227: ~0.96%9049733170104672168: ~0.96%9115912231040731994: ~0.96%9131674591042606820: ~0.96%	min: 6 tokensmean: 6.76 tokensmax: 7 tokens	min: 15 tokensmean: 18.69 tokensmax: 24 tokens

• Samples:

  anchor	rewritten_title	original_title	original_usable	product_group_id	product_id	query_id
  مانتو تابستانه سایزبزرگ	مانتو زنانه بلند کتان لمه مهتابان (Mahtaban) مدل قیطون‌دوزی شده سایزبزرگ	مانتو مزونی سایزبزرگ و بلند کتان لمه	True	7393488442012240316	p-13064828	q-77a5991bdb89f8ef0c9d74e1
  شال توری بهاره	شال رینگی زنانه بافت توری مشکی ریزش ملایم بهاره تابستانه	شال رینگی محرم بافت توری سبک بهاره تابستان	True	3428136546146965774	p-35761883	q-6df2cf14552954703bc181fe
  کاور صندلی خودرو	روکش صندلی خودرو ماتین کاور طرح لاماری جودون برای پراید 111 هاچبک رنگ عسلی	روکش صندلی جودون مخمل سنگین پراید111هاچبک طرح لاماری عسلی	False	1499769885652000917	p-22154541	q-23d3831029fb651753a41fdc

• Loss: const_learning.multi_positive.CachedMultiPositiveRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 64,
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0,
      "strict_product_unique": true,
      "positive_aggregation": "equal_mean",
      "candidate_normalization": "per_product"
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 25,835 evaluation samples
• Columns: anchor, rewritten_title, original_title, original_usable, product_group_id, product_id, and query_id
• Approximate statistics based on the first 100 samples:

  	anchor	rewritten_title	original_title	original_usable	product_group_id	product_id	query_id
  type	string	string	string	bool	int	string	string
  modality	text	text	text			text	text
  details	min: 4 tokensmean: 7.61 tokensmax: 12 tokens	min: 10 tokensmean: 22.74 tokensmax: 32 tokens	min: 4 tokensmean: 15.21 tokensmax: 38 tokens	False: ~8.65%True: ~91.35%	85480883611077928: ~0.96%94620141887068576: ~0.96%126939967775193646: ~0.96%414555575604870224: ~0.96%508123139848592225: ~0.96%705326883412186370: ~0.96%722982268963029086: ~0.96%748283295282314471: ~0.96%837572855566422369: ~0.96%879750100920247065: ~0.96%894360868531184762: ~0.96%894460498442457293: ~0.96%1157204540948018145: ~0.96%1246528872296475161: ~0.96%1295024963885167236: ~0.96%1338799806737994147: ~0.96%1377033816256841692: ~0.96%1378182285254127649: ~0.96%1523167246126709030: ~0.96%1569461136925589258: ~0.96%1656380551472137425: ~0.96%1714043282387628747: ~0.96%1821088692377293600: ~0.96%1873898255401670285: ~0.96%1887289078637752387: ~0.96%1893946278446575199: ~0.96%1897112381229869159: ~0.96%1980222813239351250: ~0.96%2178509055290690734: ~0.96%2183019638023809003: ~0.96%2274881696628349565: ~0.96%2364019919264583571: ~0.96%2585794216033412211: ~0.96%2731689991987326379: ~0.96%2794828264851915800: ~0.96%2805835676790075242: ~0.96%2981894197753378133: ~0.96%3015270045243817591: ~0.96%3239599364974634416: ~0.96%3255304357734255820: ~0.96%3331378469702228380: ~0.96%3342944947901816142: ~0.96%3408909638213825240: ~0.96%3484556783915295846: ~0.96%3668888350891467927: ~0.96%3758228385917644722: ~0.96%3809926974620475014: ~0.96%3824352126458114029: ~0.96%3859755563999554189: ~0.96%3881483706560207127: ~0.96%3939735067698965198: ~0.96%4456095283069709133: ~0.96%4461139858425942922: ~0.96%4516003944690983630: ~0.96%4592685527292491115: ~0.96%4834810160758206623: ~0.96%4973299219655891326: ~0.96%4981092919564861111: ~0.96%5004708401618002487: ~0.96%5154041703843723579: ~0.96%5175359754046346099: ~0.96%5333504649512064351: ~0.96%5383562655969689434: ~0.96%5411991990364225267: ~0.96%5449200043432178705: ~0.96%5460617467297181156: ~0.96%5506738762365109408: ~0.96%5553644345276244089: ~0.96%5555087034008504962: ~0.96%5622559576442212777: ~0.96%5636204734571133760: ~0.96%5834938150333511734: ~0.96%5876292241507693900: ~0.96%5898736471466703442: ~0.96%5911299520056911879: ~0.96%5926675982521987800: ~0.96%5935016936826697542: ~0.96%5971041036877287595: ~0.96%5978470504061304787: ~0.96%6121435259057600547: ~0.96%6159159780369340411: ~0.96%6334860989099129467: ~0.96%6468495382055659123: ~0.96%6548068253429363505: ~0.96%6566380898090667463: ~0.96%6839331460469296340: ~0.96%6943190228664021304: ~0.96%7068992290820854673: ~0.96%7282474569253430568: ~0.96%7418436171234512383: ~0.96%7525340405692444572: ~0.96%7630620700690518013: ~0.96%7668203934416791055: ~0.96%7709984240616949815: ~0.96%7816715564817642031: ~0.96%7869678632964782777: ~0.96%7930486547987510606: ~0.96%8068144898697314701: ~0.96%8098988558380748934: ~0.96%8235000430894762342: ~0.96%8361033870091434706: ~0.96%8898012224604362907: ~0.96%8926478868610738885: ~0.96%8962612547428606787: ~0.96%	min: 6 tokensmean: 6.61 tokensmax: 7 tokens	min: 15 tokensmean: 18.97 tokensmax: 24 tokens

• Samples:

  anchor	rewritten_title	original_title	original_usable	product_group_id	product_id	query_id
  رژگونه توت فرنگی	بالم لب توت فرنگی Anakan (آناکان) آبرسان ویتامینه گیاهی	بالم لب توت فرنگی آناکان	True	4981092919564861111	p-3944596	q-895b1656011116619174b327
  چای ساز کوخ	چای ساز برقی Kouch (کوخ) مدل KT-2150 مشکی، 2200 وات، کتری استیل دوجداره	چای ساز کوخ مدل KT-2150	True	3824352126458114029	p-18108877	q-dda85c17f5fd6238cab274f2
  میز کنسول	میز کنسول FARASHAHGROUP با درب طرح چوب و قفسه بندی باز	میز کنسول ارسال رایگان به مسیر های یکطرفه	True	5553644345276244089	p-15061643	q-55199bfb027445c0af320d1c

• Loss: const_learning.multi_positive.CachedMultiPositiveRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 64,
      "gather_across_devices": true,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0,
      "strict_product_unique": true,
      "positive_aggregation": "equal_mean",
      "candidate_normalization": "per_product"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 5.0
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• per_device_eval_batch_size: 256
• load_best_model_at_end: True
• dataloader_num_workers: 4

All Hyperparameters

• per_device_train_batch_size: 256
• num_train_epochs: 5.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 256
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 4.0 days
• Evaluation: 15.4 minutes
• Total: 4.0 days

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.6.0
• Transformers: 5.12.1
• PyTorch: 2.12.1+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiPositiveRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}