SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hbpkillerX/legal-clause-minilm-l6-v2")
# Run inference
sentences = [
    'Since the Balance Sheet Date, the Company Group has conducted the Business in the ordinary course consistent with past practices. Without limiting the generality of the foregoing, except as set forth on Schedule 5.13, since the Balance Sheet Date, there has not been:',
    '(a) From December 31, 2011 to the Signing Date, (i) each Parent Entity conducted such Parent Entity’s operations only in the ordinary course of business consistent with past practices and (ii) there has not occurred and continued to exist any event, change, effect, fact, circumstance or condition which, individually or in the aggregate, has had, or would reasonably be expected to have or result in, a Parent Material Adverse Change.',
    'Notwithstanding the provisions of paragraph 2 above, all Restricted Equivalents then outstanding will immediately vest, in the event of:',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9113, 0.1831],
#         [0.9113, 1.0000, 0.2245],
#         [0.1831, 0.2245, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: clause-eval-triplets
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 133,951 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 6 tokensmean: 128.79 tokensmax: 256 tokens	min: 6 tokensmean: 128.94 tokensmax: 256 tokens

• Samples:

  anchor	positive
  Except as disclosed in the SEC Documents, since December 31, 2005, there has been no material adverse change and no material adverse development in the business, properties, operations, condition (financial or otherwise), results of operations or prospects of the Company or its Subsidiaries. Since December 31, 2005, the Company has not (i) declared or paid any dividends, (ii) sold any assets, individually or in the aggregate, in excess of $50,000 outside of the ordinary course of business or (iii) had capital expenditures, individually or in the aggregate, in excess of $100,000. The Company has not taken any steps to seek protection pursuant to any bankruptcy law nor does the Company have any knowledge or reason to believe that its creditors intend to initiate involuntary bankruptcy proceedings or any actual knowledge of any fact which would reasonably lead a creditor to do so. After giving effect to the transactions contemplated hereby to occur at the Closing, the Company will not be ...	Since December 31, 1995, there has not been (a) any Material Adverse Change (as defined in Section 12.3) in the business, prospects, financial condition, revenues, expenses or operations of DAP; (b) any decrease in the cash and cash equivalents of DAP from the amounts shown on the balance sheet included in the 1995 Financial Statements (except for any such decrease attributable to any Permitted Cash Payments (as defined in Section 12.3) hereof) made by DAP prior to Closing), (c) any damage, destruction or loss, whether covered by insurance or not, having a Material Adverse Effect, with regard to DAP's properties and business; (d) any payment by DAP to, or any notice to or acknowledgment by DAP of any amount due or owing to, DAP's self-insured carrier in connection with any self-insured amounts or liabilities under health insurance covering employees of DAP, in each case, in excess of a reserve therefor on the balance sheet included in the 1995 Financial Statements; (e) any declaration,...
  Since December 31, 1995, there has not been (a) any Material Adverse Change (as defined in Section 12.3) in the business, prospects, financial condition, revenues, expenses or operations of DAP; (b) any decrease in the cash and cash equivalents of DAP from the amounts shown on the balance sheet included in the 1995 Financial Statements (except for any such decrease attributable to any Permitted Cash Payments (as defined in Section 12.3) hereof) made by DAP prior to Closing), (c) any damage, destruction or loss, whether covered by insurance or not, having a Material Adverse Effect, with regard to DAP's properties and business; (d) any payment by DAP to, or any notice to or acknowledgment by DAP of any amount due or owing to, DAP's self-insured carrier in connection with any self-insured amounts or liabilities under health insurance covering employees of DAP, in each case, in excess of a reserve therefor on the balance sheet included in the 1995 Financial Statements; (e) any declaration,...	Except as disclosed in Schedule 3(g) or -------------------------- the SEC Documents filed at least five (5) days prior to the date hereof, there has been no change or development in the business, properties, assets, operations, financial condition, results of operations or prospects of the Company or its Subsidiaries which has had or reasonably could have a Material Adverse Effect. The Company has not taken any steps, and does not currently expect to take any steps, to seek protection pursuant to any bankruptcy law nor does the Company or its Subsidiaries have any knowledge or reason to believe that its creditors intend to initiate involuntary bankruptcy proceedings.
  Except as disclosed in Schedule 3(g) or -------------------------- the SEC Documents filed at least five (5) days prior to the date hereof, there has been no change or development in the business, properties, assets, operations, financial condition, results of operations or prospects of the Company or its Subsidiaries which has had or reasonably could have a Material Adverse Effect. The Company has not taken any steps, and does not currently expect to take any steps, to seek protection pursuant to any bankruptcy law nor does the Company or its Subsidiaries have any knowledge or reason to believe that its creditors intend to initiate involuntary bankruptcy proceedings.	Except as disclosed in the SEC Documents, since December 31, 2017, there has been no material adverse change in the business, properties, operations, financial condition or results of operations of the Company or its Subsidiaries. The Company is not in violation or default of (i) any provision of the Certificate of Incorporation or Bylaws, (ii) the terms of any indenture, contract, lease, mortgage, deed of trust, note agreement, loan agreement or other agreement, obligation, condition, covenant or instrument to which it is a party or bound or to which its property is subject, or (iii) any statute, law, rule, regulation, judgment, order or decree of any court, regulatory body, administrative agency, governmental body, arbitrator or other authority having jurisdiction over the Company or any of its properties, which, in the case of clauses (ii) or (iii), could be reasonably expected to have a Material Adverse Effect. Except as described in the SEC Documents, no dispute between the Compan...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.8.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}